Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Allow option to pass device type so devices not found in scan can be included #45

Closed
madajczyk opened this issue Sep 28, 2020 · 14 comments · Fixed by #88
Closed
Labels
enhancement New feature or request

Comments

@madajczyk
Copy link

First, I'd like to say that Scrutiny is such a great idea. Thank for you for creating this!!

Feature Request
It would be awesome if there was a way to pass the device type to Scrutiny so that it can collect details on drives that the smartctl utility can't identify on it's own (due to USB bridge identification limits, etc).

Context

I deployed this in Docker to a little home server I run, and mapped in 3 devices in the config file as such:

  - device=/dev/sda
  - device=/dev/sdb
  - device=/dev/sdc

When I opened the Scrutiny UI after doing a scan, only the first two internal drives were visible. The last (/dev/sdc) is an external USB drive. I poked around and saw that it can be problematic for smartctl to get the details for external drives. However, if I could find and pass the correct device type for the USB bridge, it might work.

I was able to identify the correct value to pass for my particular drive, which allowed the retrieval of SMART data from my external drive from terminal. If I could provide configuration detail that the device is of type sat, hopefully I'll be able to monitor that external drive as well with Scrutiny. Below are the outputs of various commands I ran as I tested my theory.

sudo smartctl --scan

/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device

sudo smartctl -i /dev/sdc

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-42-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/sdc: Unknown USB bridge [0x04e8:0x6123 (0x202)]
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

sudo smartctl -i /dev/sdc -d sat

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-42-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST2000DM001-1E6164
Serial Number:    W1E86G1H
LU WWN Device Id: 5 000c50 073eb6239
Firmware Version: SC48
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Sep 27 19:10:19 2020 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
@AnalogJ
Copy link
Owner

AnalogJ commented Sep 28, 2020

This is a great idea! I actually went down the route of filtering out removable disks, since I wanted to ignore thumbdrives, but I should have thought about external hard drives connected over USB.

Let me think about this a bit. Eventually I plan on adding support for multi-host collector device grouping (and filtering) so I want to make sure that any include functionality I add is compatible with that deployment model.

@AnalogJ AnalogJ added the enhancement New feature or request label Sep 28, 2020
@develroo
Copy link

develroo commented Oct 1, 2020

It is not just removable disks. Some array controllers need parameters. so

Smartctl open device: /dev/sdf failed: DELL or MegaRaid controller, please try adding '-d megaraid,N'

optional driver flags will need to be passed especially if it is to be run in server environments (and lets face it multiple disk scenarios means servers)

Otherwise it is a great little tool and I have already installed it on non controller card servers. Cheers!

@AnalogJ
Copy link
Owner

AnalogJ commented Oct 4, 2020

Scrutiny uses smartctl --scan to try and auto-detect the correct device type. It has worked for some users with RAID controllers, but yeah in other cases smartctl just doesn't know what to do.
A config file is definitely necessary, I'm working on this off-and-on while waiting for a couple of users to verify the notifications code in a different branch is working. Once that's done I can dedicate some time to this issue.

@develroo
Copy link

develroo commented Oct 4, 2020

S'ok. I understand. Thanks for a great project and look forward to seeing improvements when they land. No worries.

Cheers.

EDIT:

FWIW I think I can see where the confusion comes in. 2 sets of devices are listed.


smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/bus/0 -d megaraid,14 # /dev/bus/0 [megaraid_disk_14], SCSI device
/dev/bus/0 -d megaraid,15 # /dev/bus/0 [megaraid_disk_15], SCSI device
/dev/bus/0 -d megaraid,18 # /dev/bus/0 [megaraid_disk_18], SCSI device
/dev/bus/0 -d megaraid,19 # /dev/bus/0 [megaraid_disk_19], SCSI device
/dev/bus/0 -d megaraid,20 # /dev/bus/0 [megaraid_disk_20], SCSI device
/dev/bus/0 -d megaraid,21 # /dev/bus/0 [megaraid_disk_21], SCSI device


I suspect that some extra parsing will be needed in these edge cases. 

@PrplHaz4
Copy link
Contributor

PrplHaz4 commented Oct 6, 2020

Scrutiny uses smartctl --scan to try and auto-detect the correct device type. It has worked for some users with RAID controllers, but yeah in other cases smartctl just doesn't know what to do.
A config file is definitely necessary, I'm working on this off-and-on while waiting for a couple of users to verify the notifications code in a different branch is working. Once that's done I can dedicate some time to this issue.

Definitely looks like lots of inconsistencies/special cases - I have an H240 card in HBA mode, and it appears that the only way --scan identifies any drives is by specifying a driver (eg. smartctl --scan -d sat). smartctl itself actually recommends using -d ccis,N for these devices, but both seem to work at the drive device level.

For this case it'd be useful to specify devices and/or drivers in the config file (similar to how you've proposed)...

smarctl --scan
<no results>

smartctl --scan -d sat
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device

@mrtumnus
Copy link

mrtumnus commented Oct 6, 2020

I'm in a similar boat here - I have a 3ware 9650 RAID card, and it requires the following form to identify the device to smartctl:
smartctl -d 3ware,<N> /dev/twa0
where <N> is the index of the drive on the RAID card

smartctl --scan doesn't automatically populate the information needed for this:

/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/sdg -d scsi # /dev/sdg, SCSI device
/dev/sdh -d scsi # /dev/sdh, SCSI device

@AnalogJ
Copy link
Owner

AnalogJ commented Oct 8, 2020

Hey,
I'm working on this issue right now. Just wanted to give you all a preview of what the collector config file will look like:

# Commented Scrutiny Configuration File
#
# The default location for this file is /scrutiny/config/collector.yaml.
# In some cases to improve clarity default values are specified,
# uncommented. Other example values are commented out.
#
# When this file is parsed by Scrutiny, all configuration file keys are
# lowercased automatically. As such, Configuration keys are case-insensitive,
# and should be lowercase in this file to be consistent with usage.


######################################################################
# Version
#
# version specifies the version of this configuration file schema, not
# the scrutiny binary. There is only 1 version available at the moment
version: 1

# This block allows you to override/customize the settings for devices detected by
# Scrutiny via `smartctl --scan`
# See the "--device=TYPE" section of https://linux.die.net/man/8/smartctl
# type can be a 'string' or a 'list'
devices:
  # example for forcing device type detection for a single disk
  - device: /dev/sda
    type: 'sat'

  # example to show how to ignore a specific disk/device.
  - device: /dev/sda
    ignore: true

  # examples showing how to force smartctl to detect disks inside a raid array/virtual disk
  - device: /dev/bus/0
    type:
      - megaraid,14
      - megaraid,15
      - megaraid,18
      - megaraid,19
      - megaraid,20
      - megaraid,21

  - device: /dev/twa0
    type:
      - 3ware,0
      - 3ware,1
      - 3ware,2
      - 3ware,3
      - 3ware,4
      - 3ware,5

Feedback is welcome & appreciated :)

@PrplHaz4
Copy link
Contributor

PrplHaz4 commented Oct 8, 2020

If the devices are being explicitly specified, I think they should be under their own key, to be used instead of or in addition to --scan results.

devices:
  - device: /dev/sda
    type: 'scsi'
  - device: /dev/sdb
    type: 'scsi'
  - device: /dev/sdc
    type: 'sat'
  - device: /dev/sdd
    type: 'sat'

I have more testing to verify this makes sense across different versions of smartctl, but wanted to get you some quick feedback.

@AnalogJ
Copy link
Owner

AnalogJ commented Oct 8, 2020

that's a good point. I'll change the keyname.

@AnalogJ
Copy link
Owner

AnalogJ commented Oct 8, 2020

Hey everyone,

I just released a beta version of the Scrutiny docker image with support for overriding the collector device detection.

The instructions for how to create the collector config file, and the new docker image tag are available in the PR description:

#88

All feedback (success & failure) is appreciated :)

@madajczyk
Copy link
Author

I tested it out with my original use case, and the external drive lit up right away:

sdc

The configuration file was dead simple to use. In the PR, I also appreciated how you set up the example.collector.yaml file with clear instructions that covered all of the use cases discussed here. Thank you very much!!

@malyyigor34
Copy link

malyyigor34 commented Jan 5, 2023

Hello. My config is:
devices:
- device: /dev/sda
type: 'scsi'
- device: /dev/sdb
type: 'scsi'
- device: /dev/sdc
type: 'sat'
- device: /dev/sdd
type: 'sat'

and i am getting error: failed to deploy a stack: services.scrutiny.devices.0 must be a string

@bertlp5
Copy link

bertlp5 commented Apr 9, 2023

and i am getting error: failed to deploy a stack: services.scrutiny.devices.0 must be a string

Had the same Problem. The Device overwrite config ist not in the docker-compose.yaml instead the device config musst be in the config folder in the file collector.yaml

@tristan-k
Copy link

tristan-k commented Jul 13, 2024

Does so the ghcr.io/analogj/scrutiny:master-omnibus docker image also allows reading the collector.yaml. I just used the example.collector.yaml as a base config and changed the name and deviced accordingly to my setup but scrutiny still recognize my /dev/sda as a ATA device event though I defined:

version: 1
host:
  id: ""
- device: /dev/sda
  type: 'SCSI'
$ smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants