ChopChop - ChopChop Is A CLI To Help Developers Scanning Endpoints And Identifying Exposition Of Sensitive Services/Files/Folders

ChopChop is a command-line tool for dynamic application security testing on web applications, initially written by the Michelin CERT.Its goal is to scan several endpoints and identify exposition of services/files/folders through the webroot. Checks/Signatures are declared in a config file (by default: chopchop.yml), fully configurable, and especially by developers."Chop chop" is a phrase rooted in Cantonese. "Chop chop" means "hurry" and suggests that something should be done now and without delay.BuildingWe tried to make the build process painless and hopefully, it should be as easy as:$ go mod download$ go build .There should be a resulting gochopchop binary in the folder.Using DockerThanks to Github Container Registry, we are able to provide you some freshly-build Docker images!docker run ghcr.io/michelin/gochopchop scan https://foobar.com -v debugBut if you prefer, you can also build it locally, see below:Build locallydocker build -t gochopchop .UsageWe are continuously trying to make goChopChop as easy as possible. Scanning a host with this utility is as simple as :$ ./gochopchop scan https://foobar.comUsing Dockerdocker run gochopchop scan https://foobar.comCustom configuration filedocker run -v ./:/app chopchop scan -c /app/chopchop.yml https://foobar.comWhat's nextThe Golang rewrite took place a couple of months ago but there's so much to do, still. Here are some features we are planning to integrate : [x] Threading for better performance [x] Ability to specify the number of concurrent threads [x] Colors and better formatting [x] Ability to filter checks/signatures to search for [x] Mock and unit tests [x] Github CI And much more!TestingTo quickly end-to-end test chopchop, we provided a web-server in tests/server.go. To try it, please run go run tests/server.go then run chopchop with the following command ./gochopchop scan http://localhost:8000 --verbosity Debug. ChopChop should print "no vulnerabilities found".There are also unit test that you can launch with go test -v ./.... These tests are integrated in the github CI workflow.Available flagsYou can find the available flags available for the scan command :FlagFull flagDescription-h--helpHelp wizard-v--verbosityVerbose level of logging-c--signaturePath of custom signature file-k--insecureDisable SSL Verification-u--url-filePath to a specified file containing urls to test-b--max-severityBlock the CI pipeline if severity is over or equal specified flag-e--exportExport type of the output (csv and/or json)--export-filenameSpecify the filename for the export file(s)-t--timeoutTimeout for the HTTP requests--severity-filterFilter Plugins by severity--plugin-filterFilter Plugins by name of plugin--threadsNumber of concurrent threadsAdvanced usageHere is a list of advanced usage that you might be interested in. Note: Redirectors like > for post processing can be used.Ability to scan and disable SSL verification$ ./gochopchop scan https://foobar.com --insecureAbility to scan with a custom configuration file (including custom plugins)$ ./gochopchop scan https://foobar.com --insecure --signature test_config.ymlAbility to list all the plugins or by severity : plugins or plugins --severity High$ ./gochopchop plugins --severity HighAbility to specify number of concurrent threads : --threads 4 for 4 workers$ ./gochopchop plugins --threads 4Ability to block the CI pipeline by severity level (equal or over specified severity) : --max-severity Medium$ ./gochopchop scan https://foobar.com --max-severity MediumAbility to specify specific signatures to be checked./gochopchop scan https://foobar.com --timeout 1 --verbosity --export=csv,json --export-filename boo --plugin-filters=Git,Zimbra,JenkinsAbility to list all the plugins$ ./gochopchop pluginsList High severity plugins$ ./gochopchop plugins --severity HighSet a list or URLs located in a file$ ./gochopchop scan --url-file url_file.txtExport GoChopChop results in CSV and JSON format$ ./gochopchop scan https://foobar.com --export=csv,json --export-filename resultsCreating a new checkWriting a new check is as simple as : - endpoint: "/.git/config" checks: - name: Git exposed match: - "[branch" remediation: Do not deploy .git folder on production servers description: Verifies that the GIT repository is accessible from the site severity: "High"An endpoint (eg. /.git/config) is mapped to multiple checks which avoids sending X requests for X checks. Multiple checks can be done through a single HTTP request. Each check needs those fields:AttributeTypeDescriptionOptional ?ExamplenamestringName of the checkNoGit exposeddescriptionstringA small description for the checkNoEnsure .git repository is not accessible from the webrootremediationstringGive a remediation for this specific "issue"NoDo not deploy .git folder on production serversseverityEnum("High", "Medium", "Low", "Informational")Rate the criticity if it triggers in your environmentNoHighstatus_codeintegerThe HTTP statu

ChopChop - ChopChop Is A CLI To Help Developers Scanning Endpoints And Identifying Exposition Of Sensitive Services/Files/Folders


ChopChop is a command-line tool for dynamic application security testing on web applications, initially written by the Michelin CERT.

Its goal is to scan several endpoints and identify exposition of services/files/folders through the webroot. Checks/Signatures are declared in a config file (by default: chopchop.yml), fully configurable, and especially by developers.



"Chop chop" is a phrase rooted in Cantonese. "Chop chop" means "hurry" and suggests that something should be done now and without delay.


Building

We tried to make the build process painless and hopefully, it should be as easy as:

$ go mod download
$ go build .

There should be a resulting gochopchop binary in the folder.


Using Docker

Thanks to Github Container Registry, we are able to provide you some freshly-build Docker images!

docker run ghcr.io/michelin/gochopchop scan https://foobar.com -v debug

But if you prefer, you can also build it locally, see below:


Build locally
docker build -t gochopchop .

Usage

We are continuously trying to make goChopChop as easy as possible. Scanning a host with this utility is as simple as :

$ ./gochopchop scan https://foobar.com

Using Docker
docker run gochopchop scan https://foobar.com

Custom configuration file
docker run -v ./:/app chopchop scan -c /app/chopchop.yml https://foobar.com

What's next

The Golang rewrite took place a couple of months ago but there's so much to do, still. Here are some features we are planning to integrate : [x] Threading for better performance [x] Ability to specify the number of concurrent threads [x] Colors and better formatting [x] Ability to filter checks/signatures to search for [x] Mock and unit tests [x] Github CI And much more!


Testing

To quickly end-to-end test chopchop, we provided a web-server in tests/server.go. To try it, please run go run tests/server.go then run chopchop with the following command ./gochopchop scan http://localhost:8000 --verbosity Debug. ChopChop should print "no vulnerabilities found".

There are also unit test that you can launch with go test -v ./.... These tests are integrated in the github CI workflow.


Available flags

You can find the available flags available for the scan command :

FlagFull flagDescription
-h--helpHelp wizard
-v--verbosityVerbose level of logging
-c--signaturePath of custom signature file
-k--insecureDisable SSL Verification
-u--url-filePath to a specified file containing urls to test
-b--max-severityBlock the CI pipeline if severity is over or equal specified flag
-e--exportExport type of the output (csv and/or json)
--export-filenameSpecify the filename for the export file(s)
-t--timeoutTimeout for the HTTP requests
--severity-filterFilter Plugins by severity
--plugin-filterFilter Plugins by name of plugin
--threadsNumber of concurrent threads

Advanced usage

Here is a list of advanced usage that you might be interested in. Note: Redirectors like > for post processing can be used.

  • Ability to scan and disable SSL verification
$ ./gochopchop scan https://foobar.com --insecure
  • Ability to scan with a custom configuration file (including custom plugins)
$ ./gochopchop scan https://foobar.com --insecure --signature test_config.yml
  • Ability to list all the plugins or by severity : plugins or plugins --severity High
$ ./gochopchop plugins --severity High
  • Ability to specify number of concurrent threads : --threads 4 for 4 workers
$ ./gochopchop plugins --threads 4
  • Ability to block the CI pipeline by severity level (equal or over specified severity) : --max-severity Medium
$ ./gochopchop scan https://foobar.com --max-severity Medium
  • Ability to specify specific signatures to be checked
./gochopchop scan https://foobar.com --timeout 1 --verbosity --export=csv,json --export-filename boo --plugin-filters=Git,Zimbra,Jenkins
  • Ability to list all the plugins
$ ./gochopchop plugins
  • List High severity plugins
$ ./gochopchop plugins --severity High
  • Set a list or URLs located in a file
$ ./gochopchop scan --url-file url_file.txt
  • Export GoChopChop results in CSV and JSON format
$ ./gochopchop scan https://foobar.com  --export=csv,json --export-filename results

Creating a new check

Writing a new check is as simple as :

  - endpoint: "/.git/config"
checks:
- name: Git exposed
match:
- "[branch"
remediation: Do not deploy .git folder on production servers
description: Verifies that the GIT repository is accessible from the site
severity: "High"

An endpoint (eg. /.git/config) is mapped to multiple checks which avoids sending X requests for X checks. Multiple checks can be done through a single HTTP request. Each check needs those fields:

AttributeTypeDescriptionOptional ?Example
namestringName of the checkNoGit exposed
descriptionstringA small description for the checkNoEnsure .git repository is not accessible from the webroot
remediationstringGive a remediation for this specific "issue"NoDo not deploy .git folder on production servers
severityEnum("High", "Medium", "Low", "Informational")Rate the criticity if it triggers in your environmentNoHigh
status_codeintegerThe HTTP status code that should be returnedYes200
headersList of stringList of headers there should be in the HTTP responseYesN/A
no_headersList of stringList of headers there should NOT be in the HTTP responseYesN/A
matchList of stringList the strings there should be in the HTTP responseYes"[branch"
no_matchList of stringList the strings there should NOT be in the HTTP responseYesN/A
query_stringGET parameters that have to be passed to the endpointStringYesquery_string: "id=FOO-chopchoptest"

External Libraries
Library NameLinkLicense
Viperhttps://github.com/spf13/viperMIT License
Go-prettyhttps://github.com/jedib0t/go-prettyMIT License
Cobrahttps://github.com/spf13/cobraApache License 2.0
strfmthttps://github.com/go-openapi/strfmtApache License 2.0
Go-homedirhttps://github.com/mitchellh/go-homedirMIT License
pkg-errorshttps://github.com/pkg/errorsBSD 2 (Simplified License)
Go-runewidthhttps://github.com/mattn/go-runewidthMIT License

Please, refer to the third-party.txt file for further information.


Talks

License

ChopChop has been released under Apache License 2.0. Please, refer to the LICENSE file for further information.


Authors
  • Paul A.
  • David R. (For the Python version)
  • Stanislas M. (For the Golang version)