Hexo Theme Feature Test - Automated Testing

This article is a complete tutorial for the matery theme’s automated testing system, paired with the other 4 feature test articles (Content Tags / Interaction & Visual / Layout & Pages / Markdown Syntax) — those 4 are the “test arena”, this one is the “test engine”.
Authoritative sources: tools/tests/release-test.sh (L1-L14), docs/superpowers/specs/2026-08-06-layout-test-cases.md.

When to Read This Article?

1. Testing Pyramid Overview

┌──────────────┐
│  L2 Prod URL  │  After release: 20 pages HTTP 200 + feature markers (curl)
│  L9/L10 SW   │  Before release: Vercel test domain SW smoke (playwright)
├──────────────┤
│  L11 Playwright Integration │  Real browser: effects/wordcloud/lazyload/read mode, 18 items
├──────────────┤
│  L14 Visual Assertions │  visual-assert precise values: 7 suites (production-blocking, skipped locally by default)
├──────────────┤
│  L13 Unit Gate │  node direct: 16 .test.js + 4 shell unit tests + 1 tutorial-structure gate
├──────────────┤
│  L1 Build Verify │  hexo g Success + artifact complete (index.html > 1000B)
└──────────────┘

2. release-test.sh Usage

# Full run (build + production checks + SW smoke + playwright integration) — release gate
tools/tests/release-test.sh

# Skip build (public/ already ready)
tools/tests/release-test.sh --skip-build

# Local iteration (tests localhost:4000, skips remote SW smoke L10 + playwright integration L11)
tools/tests/release-test.sh --local

# Skip only remote SW smoke (keep playwright integration)
tools/tests/release-test.sh --skip-l10

Base address: production is fixed at https://blog.17lai.site (port 443, never a port number); local uses LOCAL_BASE with adaptive port detection — probe order: BLOG_PORT env var → TEST_BASE env var → root _config.yml server.port (docker 4000) → 4100, take the first HTTP 200; on success exports TEST_BASE/TEST_URL, and all 36 test files follow automatically.

Execution order (156 lines of pure orchestration, not documentation order): L0 preflight → L10 → L01 → L02 → L03_05 → L07 → L08 → L09 → L11 → L12 → L2i → L13 → L13a → L14 → TC index check → summary. Note L10 runs first (before L01) — old documentation often lists it incorrectly.

Release rule: after tools/cicd.sh -r publishes, you MUST run tools/tests/release-test.sh to verify the production environment.

3. L1-L14 Layer Coverage

L11 Playwright Integration List (real browser)

4. Unit Test List (tools/tests/)

L13 Gate Unit Tests (run directly by release-test.sh)

# node unit tests (pure unit/static, no browser dependency)
unit-toc.test.js          # TOC collapse config (collapseDepth 0||6 falsy trap)
unit-search.test.js       # search three engines
unit-multilang.test.js    # multilingual config
unit-encrypt2.test.js     # encryption extensions (fragment + wiki)
unit-wiki-helper.test.js  # wiki helper
helpers.test.js           # word-count helpers (wordcount/min2read/totalcount; encrypted origin / wiki de-dup / fallback / cache reset, 52 assertions)
lang-fallback.test.js     # fallback chain 4 levels (13 assertions)
unit-i18n-core.test.js   # i18n core fallback semantics (resolveI18n 4-level chain + cfg defense, 10 cases)
unit-i18n-helper.test.js  # t() helper (fallback_lang + case tolerance, 5 cases)
echarts-charts.test.js    # ECharts charts
banner-bg.test.js         # banner & background preference linkage
homepage.test.js          # homepage structure
imgsize-unit.test.js      # image size
search-integration.test.js
infinite-scroll-aos.test.js
unit-standalone.test.js   # standalone page exclusion logic + language prefix exemption (22 assertions)
unit-markmap-prune.test.js # markmap resource pruning on pages without mindmaps (6 cases; covered by L13a_cov via unit/** glob)
unit-updated.test.js      # "Updated on" takes the git last-commit time (4 cases: CJK path / missing / overwrite as moment / author value preserved)

# shell unit tests
unit-css-architecture.test.sh   # CSS architecture (key selectors in compiled output)
unit-version.test.sh            # version increment carry (prevents 12.99→12.100)
unit-retry.test.sh              # backoff retry library
unit-deploy-target.test.sh      # deploy target switching (hexo/hexoback dual-activation verification)

Other Unit Tests (former run-tests.sh L1 / L8 sections, now merged into release-test.sh)

unit-encrypt.test.js      # self-built encryption system (AES-GCM + PBKDF2 closed loop, 9 cases)
unit-fontawesome.test.py  # FontAwesome minify script

5. External Dependency Retry & Degradation (lib/retry.sh)

Third-party network jitter is unavoidable (Algolia API / CDN videos / music APIs), and an external service being unavailable is not a product defect and should not block a release. tools/tests/lib/retry.sh handles this uniformly:

source tools/tests/lib/retry.sh

# exit-code based: backoff retry 3 times (1s/4s/9s + jitter)
retry_run "L2 search" "node tools/tests/search-integration.test.js"

# output-pattern based (used by release-test L11)
retry_match "L2d video" "node .../videobg-integration.test.js" "PASS=.*FAIL=0"

Failure classification (when all retries fail, distinguish by log signature):

6. Third-Party Interception + ERR_ABORTED Exclusion (lib/browser-env.js)

Shared environment helper for playwright tests, unifying “page’s own errors” attribution:

const { interceptThirdParty, collectPageErrors } = require('./lib/browser-env');

const env = collectPageErrors(page, BASE);   // continuously accumulates JS exceptions + same-origin resource failures
await interceptThirdParty(page);             // intercepts known pollution sources
// assert: env.jsErrors.length === 0 && env.sameOriginFailures.length === 0

7. Minify Switch Verification (cicd.sh)

tools/cicd.sh injects the minify switch per mode (sed replaces the # minify-switch marker lines in the root _config.yml):

Verification method:

# 1. debug build (unminified)
tools/cicd.sh -d
head -c 300 public/css/main.css   # multi-line, indented, with comments

# 2. minified build (-t local compile test, minified without CDN)
tools/cicd.sh -t
head -c 300 public/css/main.css   # single line, no comments, significantly smaller

After building, cicd.sh restores _config.yml (only when the changes involve just the minify-switch marker lines), keeping the working tree clean.

8. Environment Preflight, Single-Instance Lock & Base Address Adaptation

Before the gate runs, preflight (step 0) checks environment readiness; if not ready it fails immediately and prints fix commands, preventing “running halfway才发现 server is down”:

# Three fix commands printed when not ready (pick any one)
docker compose -f docker-compose.test.yml exec -T hexo bash -c 'tools/cicd.sh -d'
tools/test.sh server
docker compose -f docker-compose.test.yml exec -T hexo bash -c 'pm2 restart all'

--prepare auto-prepares: starts container if not running, rebuilds if artifacts invalid, rechecks at the end.

Single-instance lock: the gate acquires a lock at ${TMPDIR:-/tmp}/opencode/gate.lock on startup; concurrent runs are rejected outright (prints the lock-holding PID); stale locks are automatically taken over, trap EXIT releases. Concurrent gates are forbidden — they kill each other’s local server.

Base address & port adaptation: see §2’s base address section (production fixed at 443 / local port detection) — 36 test files follow automatically via exported TEST_BASE/TEST_URL.

9. L14 Visual Assertion Layer

tools/visual-assert/*.json has 16 suites total; the gate L14 only runs 7 of them: layout-checks, dark-mode, search-modal, reward-modal, homepage, about, footer.

⚠️ make visual-assert trap: that make target depends on build, which overwrites themes/matery/_config.yml placeholders in-place → triggers L5 version consistency 4-item failures. Correct usage: node tools/visual-assert.js run --local [tools/visual-assert/<suite>.json], or let the L14 layer run it.

10. TC Index & Git Hooks

The authoritative source for tests is tools/tests/**; the generated TC table lives between <!-- GENERATED:START --> and <!-- GENERATED:END --> in docs/superpowers/specs/2026-08-06-layout-test-cases.md.

First clone requires one-time setup: git config core.hooksPath .githooks. When the gate reports “TC index inconsistent”, run make tc-index then commit (bypass: git commit --no-verify).

Current index: 13 layers / 42 test files / 3 lib files / 284 assertions / 406 test names.

11. Flake Visibility & Baseline Reference Values

Flake visibility: items that pass after retry are recorded in FLAKY_LIST; the gate prints ⚠️ This gate had N flaky items (…) at the end; it does not change PASS/FAIL counts, it only makes jitter visible.

The exact PASS count grows with each version (every added assertion changes it); rely on the actual output instead of a hard-coded number.

3 libs: assert.sh (82 lines), retry.sh (114 lines, responsible for flake tracking), preflight.sh (230 lines). browser-env.js is a playwright shared helper (see §6).

12. How to Add a Test Case

Steps

1. Determine the test type:
   • Pure unit / static assertions (no browser) → L13 (node direct)
   • playwright integration (needs chromium + server) → L11
2. Create the test file tools/tests/xxx.test.js (node:test or playwright)
3. External dependency tests: wrap with retry_run / retry_match (lib/retry.sh), auto-classify and degrade on total failure
4. playwright tests: use lib/browser-env.js’s interceptThirdParty + collectPageErrors for unified error attribution
5. Register in release-test.sh at the corresponding layer (L11 or L13 section), align the output format (PASS=N FAIL=0 or N passed / 0 failed)
6. Local verification: tools/tests/release-test.sh --local (runs L13 + curl assertions); the release gate runs the full suite
7. Sync TC index: after adding/modifying, the TC index must stay in sync (.githooks/pre-commit does this automatically; or manually run make tc-index) — otherwise the gate --check reports “TC index inconsistent”

Known coverage gaps: wiki-integration.test.js and infinite-scroll-integration.test.js exist but are not hooked into any layer (pure playwright, always fails without a browser; covered by L11 or a manual pre-release playwright run). Be careful not to repeat this pattern when adding new tests.

Mocks Must Model Real Hexo Objects (Query, Not Plain Arrays)

When unit-testing with a mocked site / locals, model Hexo’s real objects — otherwise a fully green suite still lets a build-breaking bug through:

• site.posts / site.pages are Hexo Query objects (array-like): .filter() returns another Query, not an array, and they cannot be for...of-iterated — call .toArray() first.
• helpers.test.js (word count) first mocked them as plain arrays: unit tests passed, but the real build threw wikiPages is not iterable → every HTML output was 0 bytes. The mock only exercised the real path after toArray() was added.

Lesson: the closer a mock is to the real object, the better it catches build-level failures; substituting a plain array for a Query hides a missing toArray().

Code Templates

L13 unit test template (node:test, no browser):

// tools/tests/my-feature.test.js
const { describe, it } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('fs');
const path = require('path');

const PUBLIC = path.resolve(__dirname, '../../public');

describe('my-feature', () => {
  it('should render expected element', () => {
    const html = fs.readFileSync(path.join(PUBLIC, 'index.html'), 'utf8');
    assert.ok(html.includes('expected-marker'), 'missing expected marker');
  });
});

// Output at end (release-test.sh uses this line to judge pass/fail)
const pass = 1, fail = 0;
console.log(`\nResult: PASS=${pass} FAIL=${fail}`);
process.exit(fail > 0 ? 1 : 0);

L11 playwright template (needs chromium + server):

// tools/tests/my-interaction.test.js
const { chromium } = require('playwright');
const { interceptThirdParty, collectPageErrors } = require('./lib/browser-env');

const BASE = process.env.TEST_BASE || 'http://localhost:4000';
let browser, page;
let pass = 0, fail = 0;

async function setup() {
  browser = await chromium.launch();
  page = await browser.newPage();
  await interceptThirdParty(page);
}

async function testFeature() {
  const env = collectPageErrors(page, BASE);
  await page.goto(`${BASE}/posts/xxxxx/`, { waitUntil: 'networkidle' });
  const el = await page.$('.my-feature');
  if (el) { pass++; } else { fail++; console.error('FAIL: .my-feature not found'); }
  // Check no JS errors
  if (env.jsErrors.length > 0) { fail++; console.error('FAIL: JS errors', env.jsErrors); }
  else { pass++; }
}

(async () => {
  await setup();
  await testFeature();
  await browser.close();
  console.log(`\nResult: PASS=${pass} FAIL=${fail}`);
  process.exit(fail > 0 ? 1 : 0);
})();

Hooking Into the Gate: Where to Add a Line

L13 (node unit test): Append one line in the ## L13 section of release-test.sh:

run_test "my-feature" "node tools/tests/my-feature.test.js"

L11 (playwright integration): Append in the ## L11 section:

retry_match "my-interaction" "node tools/tests/my-interaction.test.js" "PASS=.*FAIL=0"

retry_match wraps external-dependency tests — auto-retries 3 times on network jitter; total failure degrades to warn instead of fail (see §5).

Worked Example: Adding a “Sidebar Collapse” Test

1. Requirement: Verify the post sidebar auto-collapses on narrow screens
2. Type: Needs browser → L11 playwright
3. Create tools/tests/sidebar-collapse.test.js (using the L11 template above)
4. Hook in: Add a retry_match line in the L11 section of release-test.sh
5. Verify: tools/tests/release-test.sh --local confirms PASS
6. TC sync: make tc-index (or let the pre-commit hook do it automatically)

Assertion Output Convention

release-test.sh judges pass/fail by output pattern, so the test file must print a matchable summary line at the end:

console.log(`\nResult: PASS=${pass} FAIL=${fail}`);
process.exit(fail > 0 ? 1 : 0);

Authoritative Test Case List

docs/superpowers/specs/2026-08-06-layout-test-cases.md (full layout test cases) + docs/testing/TESTING-PYRAMID-GUIDE.md (testing methodology) — after adding a feature, update the case list and release-test.sh coverage in sync.

13. Performance Verification (LCP Baseline, Manual)

Performance metrics (LCP) are not part of the automated gate — they require Lighthouse sampling on real devices with controlled network conditions, which assertion-style tests cannot express stably. Verification is by baseline document + reproduction commands:

• Baseline: docs/analysis/LCP-V2-BASELINE-2026-09-18.md (resource counts / Lighthouse desktop+mobile / LCP element identity / §5 reproduction commands / §7 optimization work)
• Four optimizations (2026-09-18~20): critical CSS preload · highlight conditional loading by layout · markmap localized + async · post hero-image preload
• Threshold: mobile LCP ≤ 2.5s (Search Console “40 URLs over threshold” is the observation basis)
• Pitfall: not stripping noscript inflates the resource count by 4 per page (baseline §1.3) — always strip before re-measuring
• Trigger: re-measure whenever head-level resource loading order/conditions change (CSS/JS/images)

⚠️ Known Pitfalls

Pitfall 5: Config Changes Require Clean Build

Symptom: L1 build verification passes but feature assertions fail when running the gate.

Cause: Local preview uses stale cached artifacts; test assertions are checking old behavior.

Fix: Before running the gate, you must npm run clean && npm run build, or use tools/tests/release-test.sh --prepare for auto-preparation.

Pitfall 6: source/_data/* Incremental Cache Misjudgment

Symptom: After modifying friends/gallery data, L2 URL assertions show blank pages or data doesn’t update.

Cause: Hexo’s incremental cache incorrectly judges “cached, unchanged”.

Fix: After modifying data files, run rm -f db.json && npm run build before the gate. In container environments, use tools/cicd.sh -d (includes clean).

Appendix: Test ↔ Feature Map

Purpose: Want to know “is feature X covered by automated tests?” — look up by feature name below.

Appendix: Triple-Role Self-Check List

When adding or modifying an article in the “hexo theme feature test” series, check these three items:

Quick check: If a feature “only lists its name without saying how to verify” → missing ①; “only lists syntax without showing effect” → missing ②; “doesn’t explain how to use it or is ambiguous” → missing ③.

Appendix: Testing System Quick Reference

References

系列教程

全部文章RSS订阅

Hexo 主题功能测试系列

• Hexo主题功能测试-布局与页面篇 —— 页面布局、导航、侧边栏、卡片等结构功能
• Hexo主题功能测试-交互视觉篇 —— 代码高亮、图片灯箱、动画、明暗切换等交互
• Hexo主题功能测试-内容tag篇 —— note / tabs / timeline / label / button 等 tag 插件
• Hexo主题功能测试-Markdown语法篇 —— Markdown 渲染与扩展语法
• Hexo主题功能测试-自动化测试篇 —— 主题功能的自动化测试方案