Skip to content

Commit 722b7bb

Browse files
docs-botdocs-bot
authored
Merge pull request #42232 from github/repo-sync
Repo sync
2 parents d90c371 + cd499a5 commit 722b7bb

22 files changed

Lines changed: 1411 additions & 4110 deletions

File tree

‎content/copilot/tutorials/copilot-chat-cookbook/analyze-security/manage-dependency-updates.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,8 +90,6 @@ Once you've committed the `dependabot.yml` file to your repository, {% data vari
9090

9191
### Example scenario
9292

93-
### Example scenario
94-
9593
Let's say your team has been using the basic {% data variables.product.prodname_dependabot %} configuration for a few months, and you're getting lots of dependency update pull requests. However, you're finding that:
9694

9795
* Pull requests are sitting un-reviewed because no one is automatically assigned.

‎package-lock.json‎

Lines changed: 9 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@
195195
"helmet": "^8.0.0",
196196
"highlight.js": "^11.11.1",
197197
"highlightjs-curl": "^1.3.0",
198-
"hot-shots": "^11.1.0",
198+
"hot-shots": "^12.0.0",
199199
"html-entities": "^2.5.6",
200200
"http-proxy-middleware": "^3.0.5",
201201
"imurmurhash": "^0.1.4",

‎src/article-api/README.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,3 +168,42 @@ A list of pages available for a fully qualified path containing the target langu
168168

169169
---
170170

171+
172+
## Local Development
173+
174+
### Bootstrap steps
175+
176+
1. Start the dev server:
177+
```bash
178+
npm run dev
179+
```
180+
181+
2. Test API endpoints locally:
182+
```bash
183+
curl http://localhost:4000/api/article/meta?pathname=/en/get-started
184+
curl http://localhost:4000/api/article/body?pathname=/en/get-started
185+
curl http://localhost:4000/api/pagelist/en/free-pro-team@latest
186+
```
187+
188+
### Running tests
189+
190+
```bash
191+
npm run test -- src/article-api/tests
192+
```
193+
194+
## Ownership
195+
196+
- Team: Docs Engineering
197+
198+
## Transformers
199+
200+
Currently implemented transformers:
201+
- **REST API transformer** (`rest-transformer.ts`) - Converts REST API autogenerated content
202+
- **GraphQL transformer** (`graphql-transformer.ts`) - Converts GraphQL API autogenerated content
203+
- **Audit logs transformer** (`audit-logs-transformer.ts`) - Converts audit log tables to markdown
204+
205+
### Known limitations
206+
- Some autogenerated content types don't have transformers yet
207+
- Cache invalidation is manual
208+
- No built-in rate limiting
209+
- Limited API versioning

‎src/automated-pipelines/README.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,150 @@ Slack: `#docs-engineering`
5353
Repo: `github/docs-engineering`
5454

5555
If you have a question about automation pipelines, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with one of the automation pipelines, you can open an issue in the `github/docs-engineering` repository.
56+
57+
## Sample Pipeline Template
58+
59+
### Basic pipeline structure
60+
61+
```
62+
src/<pipeline-name>/
63+
├── README.md # Pipeline documentation
64+
├── scripts/
65+
│ └── sync.ts # Main sync script
66+
├── data/ # Generated structured data (optional)
67+
│ ├── fpt/
68+
│ ├── ghec/
69+
│ └── ghes-*/
70+
├── lib/ # Utilities and helpers
71+
├── components/ # React components (if needed)
72+
├── pages/ # Next.js pages (if needed)
73+
└── tests/ # Pipeline-specific tests
74+
### Minimal sync script example
75+
76+
```typescript
77+
// scripts/sync.ts
78+
import { Command } from 'commander'
79+
80+
const program = new Command()
81+
.description('Sync <pipeline-name> data')
82+
.option('--source-branch <branch>', 'Source repo branch', 'main')
83+
.parse()
84+
85+
const opts = program.opts()
86+
87+
async function main() {
88+
// 1. Fetch data from external source
89+
// 2. Transform data
90+
// 3. Write to data/ directory (or generate Markdown)
91+
// 4. Validate output
92+
}
93+
94+
main()
95+
```
96+
97+
### Workflow example
98+
99+
```yaml
100+
# .github/workflows/sync-<pipeline-name>.yml
101+
name: Sync <pipeline-name>
102+
103+
on:
104+
workflow_dispatch:
105+
inputs:
106+
SOURCE_BRANCH:
107+
description: 'Branch to sync from'
108+
default: 'main'
109+
schedule:
110+
- cron: '16 20 * * *' # Daily
111+
112+
jobs:
113+
sync:
114+
runs-on: ubuntu-latest
115+
steps:
116+
- uses: actions/checkout@v4
117+
- run: npm ci
118+
- run: npm run sync-<pipeline-name>
119+
```
120+
121+
## Ownership Table
122+
123+
| Pipeline | Owning Team | Source Data Owner | Sync Frequency |
124+
|----------|-------------|-------------------|----------------|
125+
| REST | Docs Engineering | API Platform | Daily |
126+
| GraphQL | Docs Engineering | API Platform | Daily |
127+
| Webhooks | Docs Engineering | API Platform | Daily |
128+
| CodeQL CLI | Docs Engineering | Code Scanning | Per release |
129+
| GitHub Apps | Docs Engineering | Integrations | Daily |
130+
| Audit Logs | Docs Engineering | Enterprise | Daily |
131+
| Secret Scanning | Docs Engineering | Security | Daily |
132+
133+
## Migration Status
134+
135+
### Active pipelines (✅ Production)
136+
- REST API - Fully automated, daily sync
137+
- GraphQL API - Fully automated, daily sync
138+
- Webhooks - Fully automated, daily sync
139+
- GitHub Apps - Fully automated, daily sync
140+
- Secret Scanning - Fully automated, daily sync
141+
- Audit Logs - Fully automated, daily sync
142+
143+
### Manual sync per release
144+
- CodeQL CLI - Manual sync per release
145+
146+
### Legacy pipelines (📦 To migrate)
147+
- None currently identified
148+
149+
### Migration patterns
150+
151+
When migrating manual content to automated pipelines:
152+
1. **Audit existing content** - Document current structure
153+
2. **Source data analysis** - Identify gaps between source and docs
154+
3. **Data enrichment** - Work with source team to add missing fields
155+
4. **Scraping phase** - Temporarily scrape existing content to preserve prose
156+
5. **Gradual migration** - Migrate section by section
157+
6. **Validation** - Compare old vs new output
158+
7. **Deprecate manual** - Remove manual content once automated is stable
159+
160+
## Shared Components
161+
162+
Components in `src/automated-pipelines/components/` available for reuse:
163+
- Parameter tables
164+
- Response schemas
165+
- Code example formatting
166+
- Common layout patterns
167+
168+
## Testing Strategy
169+
170+
### Automated pipeline tests
171+
172+
All autogenerated pages tested in `src/automated-pipelines/tests/rendering.ts`:
173+
- Page renders without errors
174+
- Required sections present
175+
- Links are valid
176+
- Schema validation
177+
178+
### Pipeline-specific tests
179+
180+
Each pipeline in `src/<pipeline-name>/tests/` should test:
181+
- Data transformation logic
182+
- Schema validation
183+
- Version handling
184+
- Edge cases specific to that pipeline
185+
186+
### Testing locally
187+
188+
```bash
189+
# Run all automated pipeline tests
190+
npm run test -- src/automated-pipelines/tests
191+
192+
# Run specific pipeline tests
193+
npm run test -- src/<pipeline-name>/tests
194+
```
195+
196+
We are not expecting significant investment here, but we will add and support pipelines as needed to meet business needs.
197+
198+
### Developer experience
199+
- Pipeline scaffolding tool
200+
- Validation helpers
201+
- Testing fixtures
202+
- Documentation generator

‎src/codeql-cli/README.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,79 @@ Slack: `#docs-engineering`
4646
Repo: `github/docs-engineering`
4747

4848
If you have a question about the CodeQL CLI pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the CodeQL CLI pipeline, you can open an issue in the `github/docs-engineering` repository.
49+
50+
## Dependency Matrix
51+
52+
### Required dependencies
53+
54+
| Dependency | Purpose | Installation | Notes |
55+
|------------|---------|--------------|-------|
56+
| **Pandoc** | Convert ReStructuredText to Markdown | `brew install pandoc` (macOS) | Version 2.x+ required |
57+
| **semmle-code repo** | Source ReStructuredText files | Clone at root of docs-internal | Private GitHub repo |
58+
| **Node.js** | Run sync script | Already in project | Version specified in `.nvmrc` |
59+
60+
### Source data location
61+
- **Repository**: `github/semmle-code` (private)
62+
- **Branch**: Configurable (default: `main`)
63+
- **Source files**: ReStructuredText documentation in semmle-code
64+
- **Output location**: `content/code-security/codeql-cli/codeql-cli-manual/`
65+
66+
### Version compatibility
67+
- CodeQL CLI releases approximately every 2 weeks
68+
- Pipeline should be run to align with CLI releases
69+
- Multiple CLI versions may be documented simultaneously
70+
71+
## Escalation Policy
72+
73+
### Ownership
74+
- **Team**: Docs Engineering
75+
- **Source team**: Code Security (CodeQL team)
76+
77+
### Escalation path
78+
1. **Pipeline failures** → #docs-engineering Slack
79+
2. **Source data issues** → #code-security-docs Slack
80+
3. **Pandoc conversion issues** → Check Pandoc version and file format
81+
4. **Repo access issues** → Request access to `github/semmle-code`
82+
83+
### On-call procedures
84+
If the pipeline fails:
85+
1. Check workflow logs for errors
86+
2. Verify Pandoc is installed and working
87+
3. Confirm access to `semmle-code` repo
88+
4. Check for ReStructuredText format changes
89+
5. Test conversion locally before re-running
90+
6. Escalate to Code Security team if source data issue
91+
92+
### Common failure modes
93+
- **Pandoc not found** - Install Pandoc locally or in CI
94+
- **semmle-code not cloned** - Clone repo at docs-internal root
95+
- **Conversion errors** - Check for unsupported RST syntax
96+
- **Branch not found** - Verify branch name in workflow input
97+
98+
## Known Constraints
99+
100+
### Pipeline limitations
101+
- **Manual trigger only** - No automated schedule (runs on-demand)
102+
- **Pandoc dependency** - Requires external tool installation
103+
- **Repo access** - Requires access to private `semmle-code` repo
104+
- **Local testing** - Must clone semmle-code locally to test
105+
106+
### Content constraints
107+
- **Frontmatter preservation** - Only first sync sets frontmatter
108+
- **Introduction placement** - Must be above auto-generation comment
109+
- **No inline editing** - Cannot edit autogenerated sections
110+
- **Version handling** - Limited liquid versioning support
111+
112+
### Technical constraints
113+
- **ReStructuredText format** - Source must be valid RST
114+
- **Pandoc conversion** - Limited control over conversion output
115+
- **File naming** - Determined by source file structure
116+
- **Link rewriting** - May need manual adjustment after generation
117+
118+
### Process constraints
119+
- **Release alignment** - Sync timing depends on CodeQL releases
120+
- **PR review required** - Changes must be reviewed before merge
121+
- **Content coordination** - Writers must coordinate intro additions
122+
- **Multiple versions** - May need separate syncs for version branches
123+
124+
We are not expecting significant new work to this pipeline, but we will support incoming changes from semmle-code.

0 commit comments

Comments
 (0)