feat: output security reports as JSON when requested

[jajanet]

This PR adds the ability to output security report results in JSON. This enables programmatic parsing for accuracy checks, standardization, and integration with SCM tools and CI/CD pipelines (e.g., GitHub Actions, Jenkins)

Example of original markdown report and corresponding JSON:

- **Vulnerability:** Path Traversal and Command Injection
- **Vulnerability Type:** Security
- **Severity:** Critical
- **Source Location:** `lib/router.js`
- **Line Content:** `full_path = "" + dispatch.static_route + (unescape(pathname));`
- **Description:** The `pathname` variable, derived from the URL, is not sanitized before being used to construct a file path. An attacker can use URL-encoded characters like `../` to traverse the file system and access arbitrary files. This vulnerability is further escalated to command injection because the `full_path` is used in a `spawn` call, allowing an attacker to execute arbitrary commands on the system.
- **Recommendation:** Sanitize the `pathname` variable by removing any directory traversal characters before using it to construct a file path. Use `path.normalize()` or a similar function to resolve the path and ensure it stays within the intended directory.

turns into

[
    {
        "vulnerability": "Path Traversal and Command Injection",
        "vulnerabilityType": "Security",
        "severity": "Critical",
        "extension": {
            "sourceLocation": {
                "File": "lib/router.js",
                "startLine": null,
                "endLine": null
            },
            "sinkLocation": {
                "File": null,
                "startLine": null,
                "endLine": null
            },
            "dataType": null
        },
        "lineContent": "`full_path = \"\" + dispatch.static_route + (unescape(pathname));`",
        "description": "The `pathname` variable, derived from the URL, is not sanitized before being used to construct a file path. An attacker can use URLencoded characters like `../` to traverse the file system and access arbitrary files. This vulnerability is further escalated to command injection because the `full_path` is used in a `spawn` call, allowing an attacker to execute arbitrary commands on the system.",
        "recommendation": "Sanitize the `pathname` variable by removing any directory traversal characters before using it to construct a file path. Use `path.normalize()` or a similar function to resolve the path and ensure it stays within the intended directory."
    },
...
]

Fields are optional and written as null if not present, as the tool assumes that the DRAFT_SECURITY_REPORT.md file is well-formed and has the expected fields

This is an initial implementation to help improve processes, and we may iterate using Vertex calls in the future. There is an upcoming PR on adding a subfield for code changes sometimes present under recommendation as well

[gemini-code-assist]

Summary of Changes

Hello @jajanet, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the security analysis tool by introducing the ability to output findings in a structured JSON format. This change is crucial for enabling automated processing, integration with external systems like CI/CD pipelines, and improving the overall utility of the security reports. The implementation involves adding a new parsing utility, integrating it as a server tool, and updating the workflow configuration to support conditional JSON generation.

Highlights

• JSON Output for Security Reports: Introduced the capability to generate security analysis reports in JSON format, alongside the existing Markdown output. This feature is activated when explicitly requested by the user (e.g., via --json flag).
• New Markdown Parsing Logic: Added a new parser.ts module containing parseMarkdownToDict function, responsible for converting the structured Markdown security reports into a standardized JSON object array. This includes handling various field formats and extracting location details.
• Integration of JSON Conversion Tool: A new convert_report_to_json tool has been registered in mcp-server/src/index.ts. This tool reads the Markdown report, utilizes the new parsing logic, and writes the resulting JSON to .gemini_security/security_report.json.
• Updated Workflow Configuration: The analyze.toml configuration has been updated to conditionally invoke the convert_report_to_json tool during the final reporting phase and to include a 'Generate JSON report' task in the initial plan if JSON output is requested.
• Comprehensive Unit Tests for Parser: A new test file parser.test.ts has been added, providing extensive unit tests for the parseMarkdownToDict function to ensure accurate and robust parsing of various Markdown report structures, including handling missing data and different formatting.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• commands/security/analyze.toml
  • Modified the 'Final Reporting & Cleanup' phase to include a conditional call to the convert_report_to_json tool if JSON output is requested.
  • Added a new action to inform the user about the availability of the JSON report at .gemini_security/security_report.json.
  • Updated the initial plan generation to include a 'Generate JSON report' task when JSON output is requested.
• mcp-server/src/index.ts
  • Imported the new parseMarkdownToDict function.
  • Registered a new server.tool named convert_report_to_json to handle the conversion of Markdown security reports to JSON.
  • The convert_report_to_json tool reads the Markdown report, parses it, and writes the JSON output to a file.
• mcp-server/src/parser.test.ts
  • Added a new file containing unit tests for the parseMarkdownToDict function.
  • Tests cover various scenarios including standard vulnerabilities, privacy violations, multiple findings, markdown formatting, missing line numbers, and missing sink locations.
• mcp-server/src/parser.ts
  • Added a new file defining Location and Finding interfaces for the structured JSON output.
  • Implemented parseLocation function to extract file paths and line numbers from location strings.
  • Implemented parseMarkdownToDict function to parse Markdown security reports into an array of Finding objects, handling different field formats and markdown syntax.

Activity

• The pull request was created by jajanet with the clear intent of adding JSON output capabilities for security reports.
• New files mcp-server/src/parser.ts and mcp-server/src/parser.test.ts were added to implement and test the Markdown to JSON parsing logic.
• Existing files commands/security/analyze.toml and mcp-server/src/index.ts were modified to integrate the new JSON output feature into the existing workflow.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

🤖 Hi @jajanet, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[gemini-code-assist]

Code Review

This pull request introduces a valuable feature to output security reports in JSON format, which will greatly improve programmatic consumption of the results. The implementation includes a new markdown parser, tests, and a server tool to handle the conversion.

My review focuses on improving the robustness and correctness of the new parser and its integration. I've identified a critical bug in the parsing logic that could lead to incorrect JSON output, along with a few medium-severity issues related to error handling, a typo in a prompt, and code consistency. The provided code suggestions aim to resolve these issues.

[github-actions]

📋 Review Summary

This pull request introduces a valuable feature to output security reports in JSON format. The implementation is well-executed, with a robust markdown parser and comprehensive test coverage. The code is clean, and the new functionality is a great addition for enabling programmatic use of the security reports.

🔍 General Feedback

• I've left a few minor suggestions for improving type safety and code readability, but overall this is a solid contribution.

[github-actions]

🟡 Casting the function to `any` using `as any` bypasses TypeScript's type safety checks. It would be more robust to define and use a specific type for the server tool implementation to ensure type safety and prevent potential runtime errors.

[github-actions]

🟢 The regular expression for extracting fields is quite complex. Adding a comment to explain the different parts of the regex would improve readability and make it easier for future maintainers to understand and modify the code. For example, a brief explanation of the lookahead used to delimit fields would be very helpful.

[github-actions]

🟢 This is a well-crafted regular expression for splitting the report into individual vulnerability sections. The use of a positive lookahead `(?=...)` to split by "Vulnerability:" while keeping the delimiter is a great approach. It also handles optional markdown headings, which makes the parsing more robust.

[github-actions]

## 📋 Security Analysis Summary

This pull request introduces a new feature to output security reports in JSON format. The implementation adds a new tool to the MCP server and a parser to convert the markdown report to JSON. The code is well-structured and includes tests for the new parser.

🔍 General Feedback

• The new feature is a valuable addition that will improve the usability of the security extension.
• The code is clean and easy to understand.
• One potential security vulnerability was identified in the new parser.

[github-actions]

High ReDoS in parser

The regular expression used to parse the markdown report is vulnerable to Regular Expression Denial of Service (ReDoS). An attacker can craft a malicious markdown file that will cause the regex engine to backtrack excessively, leading to a denial of service. The vulnerable part is ([\\s\\S]*?) which can cause catastrophic backtracking.

Suggested change

	const pattern = new RegExp(patternStr, 'i');
	const fieldNames = 'Vulnerability Type|Severity|Source Location|Sink Location|Data Type|Line Content|Description|Recommendation';
	const patternStr = `(?:-?\\\\s*\\\\**)?${label}\\\\**:\\\\s*([\\\\s\\\\S]*?)(?=\\\\n(?:-?\\\\s*\\\\**)?(?:${fieldNames})|$)`;
	const pattern = new RegExp(patternStr, 'i');

[shrishabh]

can we fix this?

[QuanZhang-William]

Thank you for the PR!

Left some questions and I think we should also update the README.md

[shrishabh]

Thank you for the PR!

I question that I was wondering - here we are deterministically parsing the markdown into json using regexes. Another option could have been using LLMs to adjust their outputs so that they conform to this particular format. I think deterministically parsing is a great first step, but we might want to keep an eye out for failure cases and potentially move to the alternative if we see lots of errors.

[shrishabh]

can we fix this?

[shrishabh]

What happens if there is an error in any of these extraction methods? It might be worth considering failing gracefully here as the rest of the extraction might still produce a reasonable report?

[QuanZhang-William]

/lgtm to me.

Please address other feedbacks before merge :)