I am having trouble getting CVE scan data since the cve scan has been moved to vulners.py

Is there documentation on this? If so, I will look it over to see if I can figure out what is wrong.

If not here is the relevant data:

[root@hostname cve]# pwd
/srv/salt/hubblestack_data/hubblestack_nova_profiles/cve
[root@hostname cve]# cat vulners.yaml
vulners_scanner: True
vulners_api_key: <my key>

pwd; cat top.nova
/srv/salt/hubblestack_data/hubblestack_nova_profiles
# Default top.nova
#
# Subscribes to CIS, cve_scan, and misc.yaml for miscellaneous checks

nova:
  '*':
    #- security.meltdown_spectre
    - security.ssh_passwordauthentication
    #- cis.distribution-independent-linux-level-1-all-v1-1-0
    - vulners

Output from hubble audit:

hubble hubble.audit
{'Compliance': '0%',
 'Failure': [{'sshd-authenticationmethods-publickey': 'Check for explicitly configured publickey authentication method'},
             {'sshd-passwordauthentication-no': 'Ensure password authentication is disabled in sshd_config'}]}

Also, I would like to make sure the functionality in the previous scan to be able to use a local json file for CVE data was still supported.

salt pj-rhel6-sensoring.lab04.local hubble.audit OS_Linux . This is only on RHEL6 VM . We have other linux VM ( Centso6 , Cetos7 , RHEL7 ) but error is specific to RHEL6 which is on Red Hat Enterprise Linux Server release 6.9 (Santiago) .

pj-rhel6-sensoring.lab04.local:
    ----------
    Compliance:
        68%
    Errors:
        |_
          ----------
          **/modules/stat_nova.py**:
              ----------
              data:
                  CommandExecutionError: Path not found: /usr/local/patchagent/patchservice
              error:
                  exception occurred
    Failure:
        |_
          ----------
          OS_Linux_v12-02:
              Ensure timezone is set correctly
        |_
          ----------
          OS_Linux_v12-15:
              SSH Root login disabled
        |_
          ----------
          OS_Linux_v12-16:
              Ensure iptables modules are loaded
        |_
          ----------
          OS_Linux_v12-10:
              Ensure sudoers file has non-default commands added to it
        |_
          ----------
          OS_Linux_v12-03:
              Ensure patchagent services are running
        |_
          ----------
          OS_Linux_v12-05:
              Ensure swap volume is on separate disk
    Success:
        |_
          ----------
          OS_Linux_V12-11:
              Root passwd should be set
        |_
          ----------
          OS_Linux_v12-13:
              No blank passwords allowed



**salt-minion version**  
[root@pj-rhel6-sensoring tmp]# salt-minion -V
Salt Version:
           Salt: 2017.7.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.14 (default, Jan 31 2018, 02:12:13)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: redhat 6.9 Santiago
         locale: UTF-8
        machine: x86_64
        release: 2.6.32-696.13.2.el6.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 6.9 Santiago

oval_scanner.py has been refactored to provide more accurate package version comparison and includes some fixes. It also captures greater details about advisories, and splunk_nova_return.py has been refactored to send those details to a Splunk backend. For comparison, this is how data in Splunk looks before the refactor:

{ 
   check_id: RHSA-2015:1705: bind security update (Important)
   check_result: Failure
   description: Vulnerable Package(s): bind-libs-32:9.11.4-9.P2.el7, bind-libs-lite-32:9.11.4-9.P2.el7, bind-license-32:9.11.4-9.P2.el7, bind-utils-32:9.11.4-9.P2.el7
   dest_fqdn: <hostname>
   dest_host: <hostname>
   dest_ip: <ip address>
   job_id: <job_id>
   minion_id: <minion id>
   system_uuid: <system uuid>
}

After the refactor, the data looks like this:

{ 
   advisory: { 
     RHSA-2020:2344: https://access.redhat.com/errata/RHSA-2020:2344
   }
   check_id: RHSA-2020:2344: bind security update (Important)
   check_result: Failure
   cve: [ 
     { 
       CVE-2020-8616: https://access.redhat.com/security/cve/CVE-2020-8616
     }
     { 
       CVE-2020-8617: https://access.redhat.com/security/cve/CVE-2020-8617
     }
   ]
   description: Vulnerable Package(s): bind-export-libs-32:9.11.4-9.P2.el7, bind-libs-32:9.11.4-9.P2.el7, bind-libs-lite-32:9.11.4-9.P2.el7, bind-license-32:9.11.4-9.P2.el7, bind-utils-32:9.11.4-9.P2.el7
   dest_fqdn: <hostname>
   dest_host: <hostname>
   dest_ip: <ip address>
   impated_pkgs:
     {
       name: bind-export-libs
       version: 32:9.11.4-9.P2.el7
     }
     { 
       name: bind-libs
       version: 32:9.11.4-9.P2.el7
     }
     { 
       name: bind-libs-lite
       version: 32:9.11.4-9.P2.el7
     }
     {
       name: bind-license
       version: 32:9.11.4-9.P2.el7
     }
     {
       name: bind-utils
       version: 32:9.11.4-9.P2.el7
     }
   ]
   job_id: <job id>
   minion_id: <minion id>
   severity: Important
   system_uuid: <system uuid>
}

The more detailed data gives us the ability to make better dashboards within Splunk, as well as pivot off the specifics to give us better flexibility in reporting.

Along with this, the oval_scanner.py vulnerability collections are now threaded for faster reporting.

Hey there @basepi

We met briefly at the Adobe Open Source Summit and I said I was interested in creating you a logo for this project. I think I remember you asking for something flattish design? In any case, I've come up with an idea attached here. Let me know if this is something that you're looking for.

Large:

Small:

This module is capable of connecting to any port and fetching certificate details for a certificate attached on it. The module is supposed to work in conjunction with fdg's osquery module. The osquery module would supply information about open ports after which this module extracts certificate information.

I was talking with George Starcher about a minor issue in hubblestack/splunklogging's hec object. I fixed that minor bug, but later learned sendEvent() isn't actually used (just batch events).

While trying to fix the non-issue I noticed there were four versions of his http_event_collector object and four ways to fetch the options for it. I've attempted to DRY those objects and their _get_options for easier ongoing maintenance. (Suppose the next bug is real, but now we have 4 variants of the HEC to modify.)

Thoughts welcome.

For the record, the Elusive Signing Bug is a subtle bug where when a file fails the repo signing checks, it's still sometimes served to the code that was looking for it. It was extremely difficult to reproduce for some reason (even though some people could reproduce it every time).

I spent a horrific amount of time tracking this down. I finally determined that cp.cache_dir was never meant to be a proper sync. The original authors of hubble.audit.sync provided a clean argument that would file.remove the cache dir to prevent objects deleted from the repo sticking around after being elided.

The problem is that removing the whole cache and re-downloading it on every schedule loop is expensive, so someone disabled it later. It was probably thought that the regular file_client.channel.fs.update() (which invokes the fileserver.reap_fileserver_cache_dir()) would properly handle these cases; but it definitely does not appear to do so.

Essentially the problem is the double cache copy situation:

1. we first copy from the fileserver (roots, or gitfs or whatever) into roots (i.e. /var/cache/hubble/roots)
2. cp.cache_dir then copies from roots to files (i.e. /var/cache/hubble/files)
3. the daemon file_client.channel.fs.update() updates the roots (but never attempts to clean up the files copy).
4. signing needs these files to not be found, which is handled in roots but not files

It's worth noting that the unwelcome leftover files were at least downloaded in good faith. To get downloaded at all, they either had to be downloaded before signing was a thing or at some point when signing verified they were OK...

But they should definitely get cleaned up when they fail the signing pass.

The fix: I taught cp.cache_dir how to notice these files missing in roots that exist in files. It can now remove them automatically during what should have been a proper sync. Note that the old behavior can be restored by setting cleanup_existing=False but I can't think of a scenario where this would be desirable.

There's still the remaining question: Why did this affect hubble.audit but not audit.run?

Answer: hubble.audit uses cp.cache_dir to copy the whole thing all at once; but audit.run uses cp.cache_file and checks the result. Specifically checking one file, will indeed reveal the file is missing in roots even if it exists in files as a spurious older file. (module_runner.runner.make_file_available also uses cp.cache_file, so the hubble.audit issue is entirely avoided in the successor).

my old (wrong) analysis: ~~Basically the problem is that various parts of the hubble code base read from the fileserver cache directly (rather than using the intended interfaces for such tasks). The repo signing checks are injected into the fileserver find_file() mechanisms, which normally trigger a fileserver.reap_cache_dir() hit, causing the file to be deleted from the cache.~~

~~But in some cases, apparently, a cache refresh isn't even part of the invocation of the execution module(s) in question. So the "reaping" never happens (at least not during the execution) and the failing file is serv^H^H^H^H read from the cache as if it was OK.~~

~~All this means, that it feels like a race condition to me. Eventually there would have been a cache refresh that would have cleared the failing file (via the reaping mechanism). This patch simply forces the reaping of files that fail the signing check so there's no need to wait for the reaping and no question about when it happens in the case of a signature check failure.~~

Apparently we do want to extend the sourcetype name from hubble_fdg to hubble_fdg_filename but let's choose to do that without the salt:// protocol and without the .fdg file extension; while we're at it, make sure to replace all non alphanumerics with an underscore.

**# salt '*' hubble.sync minion1: 'hubble.sync' is not available. minion2: 'hubble.sync' is not available.

salt '*' hubble.audit

minion1: 'hubble.audit' is not available. minion2: 'hubble.audit' is not available.**

We are using hubble module for audit and it is wokring fine for RHEL and Centos but on SLES linux hubble.sycn is not working. For SLES the official salt-minion version is salt-minion-2016.11.4-13.1.x86_64 . We do not have any updated version for SLES11 SP4.

Trouble shooting

salt '' saltutil.clear_cache salt '' saltutil.sync_all salt '*' hubble.sync

tried above commands still no luck for SLES11 SP4 minions.

**Salt master version

# salt-master --versions-report Salt Version: Salt: 2017.7.2

Dependency Versions: cffi: Not Installed cherrypy: unknown dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.7.2 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.5 (default, Aug 4 2017, 00:39:18) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4

System Versions: dist: centos 7.4.1708 Core locale: UTF-8 machine: x86_64 release: 3.10.0-693.11.1.el7.x86_64 system: Linux version: CentOS Linux 7.4.1708 Core

salt minion version on SLEL11 SP4

# /usr/bin/salt-minion --versions-report Salt Version: Salt: 2016.11.4

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.6 libgit2: Not Installed libnacl: Not Installed M2Crypto: 0.21.1 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.6.9 (unknown, Apr 7 2015, 08:28:12) python-gnupg: Not Installed PyYAML: 3.10 PyZMQ: 14.0.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.0.8

System Versions: dist: SuSE 11 x86_64 machine: x86_64 release: 3.0.101-68-default system: Linux version: SUSE Linux Enterprise Server 11 x86_64

hubble module is missing on SLES11SP4 minions while checking with sys.list_modules but on Centos6 and Centos7 I can see the hubble module without any issue.

salt 'minion1' sys.list_modules

minion1: - acl - aliases - alternatives - appcontrol - archive - artifactory - at - beacons - blockdev - bridge - btrfs - buildout - certificates_import - cloud - cmd - composer - config - consul - container_resource - cp - cpan - cron - data - defaults - devmap - dig - disk - django - dnsmasq - dnsutil - drbd - elasticsearch - environ - etcd - ethtool - event - extfs - file - gem - genesis - grains - group - grub - hashutil - hipchat - hosts - http - img - incron - ini - inspector - introspect - ip - iptables - iwtools - jboss7 - jboss7_cli - k8s - key - kmod - locale - locate - logrotate - lowpkg - lvm - match - mine - minion - modjk - mount - nagios_rpc - network - nfs3 - nova_loader - openscap - openstack_config - oscap - pagerduty - pam - partition - pillar - pip - pkg - pkg_resource - postfix - postgres_cfg - ps - publish - puppet - pushover - pyenv - quota - raid - random - random_org - rbenv - rest_sample_utils - restartcheck - ret - rsync - rvm - s3 - s6 - salt_proxy - saltutil - schedule - scsi - sdb - seed - sensors - service - shadow - slack - slsutil - smbios - smtp - sqlite3 - ssh - state - status - supervisord - sys - sysctl - sysfs - syslog_ng - system - temp - test - timezone - tomcat_cfg - udev - user - vbox_guest - virtualenv - x509 - xfs

There probably isn't ever a good time to tell Splunk to index extra fields.

The changes to the indexed fields intended by the removed code have to be accompanied by changes to the fields configuration in Splunk or oddball problems will occur (e.g., duplicate field indexing). The best plan (if you really really need extra index fields) is to ask the Splunk admins to add them to the index through their usual methods.

This affords them the opportunity to say, "that's not a good idea because ____" and prevents the data from getting wonky in their indexes.

There are certainly fields one might wish to add to the index. It's best to coordinate them carefully with the Splunk admins. (And these fields are more rare than you'd think if you're accustomed to traditional databases.)

From @jrporcaro on July 6, 2017 22:34

I am using salt 2017.7.0rc1 (Nitrogen) with a default install of hubble 2017.4.1 on CentOS 7.2 using pygit2 and gitfs based hubble install. I did a saltutil.sync_all then a hubble.sync then a hubble.audit.

[root@master2 ~]# salt \* hubble.audit
master2:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1466, in _thread_return
        return_data = executor.execute()
      File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 28, in execute
        return self.func(*self.args, **self.kwargs)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 108, in audit
        show_compliance=show_compliance)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 406, in top
        load()
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 567, in load
        __nova__ = NovaLazyLoader()
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 667, in __init__
        self._load_all()
      File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1611, in _load_all
        self._load_module(name)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 828, in _load_module
        module_name,
    ValueError: too many values to unpack

here is my versions_report:

Salt Version:
           Salt: 2017.7.0rc1

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.24.6
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.2
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-327.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core

Copied from original issue: hubblestack/hubble-salt#83