gitea/tests/integration
Jason Song 1e76a824bc
Refactor and enhance issue indexer to support both searching, filtering and paging (#26012)
Fix #24662.

Replace #24822 and #25708 (although it has been merged)


## Background

In the past, Gitea supported issue searching with a keyword and
conditions in a less efficient way. It worked by searching for issues
with the keyword and obtaining limited IDs (as it is heavy to get all)
on the indexer (bleve/elasticsearch/meilisearch), and then querying with
conditions on the database to find a subset of the found IDs. This is
why the results could be incomplete.

To solve this issue, we need to store all fields that could be used as
conditions in the indexer and support both keyword and additional
conditions when searching with the indexer.

## Major changes

- Redefine `IndexerData` to include all fields that could be used as
filter conditions.
- Refactor `Search(ctx context.Context, kw string, repoIDs []int64,
limit, start int, state string)` to `Search(ctx context.Context, options
*SearchOptions)`, so it supports more conditions now.
- Change the data type stored in `issueIndexerQueue`. Use
`IndexerMetadata` instead of `IndexerData` in case the data has been
updated while it is in the queue. This also reduces the storage size of
the queue.
- Enhance searching with Bleve/Elasticsearch/Meilisearch, make them
fully support `SearchOptions`. Also, update the data versions.
- Keep most logic of database indexer, but remove
`issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is
the entry point to search issues.
- Start a Meilisearch instance to test it in unit tests.
- Add unit tests with almost full coverage to test
Bleve/Elasticsearch/Meilisearch indexer.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-07-31 06:28:53 +00:00
..
migration-test Rewrite logger system (#24726) 2023-05-21 22:35:11 +00:00
schemas
actions_trigger_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
admin_config_test.go Fix admin config page error, use tests to cover the admin config and 500 error page (#24965) 2023-05-29 15:00:21 +00:00
admin_user_test.go
api_actions_artifact_test.go Actions Artifacts support uploading multiple files and directories (#24874) 2023-07-21 10:42:01 +08:00
api_activitypub_person_test.go Refactor web package and context package (#25298) 2023-06-18 09:59:09 +02:00
api_admin_org_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_admin_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
api_branch_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_comment_attachment_test.go For API attachments, use API URL (#25639) 2023-07-10 09:31:19 +00:00
api_comment_test.go For API attachments, use API URL (#25639) 2023-07-10 09:31:19 +00:00
api_feed_user_test.go
api_fork_test.go
api_gitignore_templates_test.go
api_gpg_keys_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_helper_for_declarative_test.go
api_httpsig_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_attachment_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_config_test.go
api_issue_label_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_milestone_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_pin_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_reaction_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_stopwatch_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_subscription_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_issue_test.go Add commits dropdown in PR files view and allow commit by commit review (#25528) 2023-07-28 21:18:12 +02:00
api_issue_tracked_time_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_keys_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_label_templates_test.go Add API for Label templates (#24602) 2023-05-23 18:10:23 +08:00
api_license_templates_test.go
api_nodeinfo_test.go Add commits dropdown in PR files view and allow commit by commit review (#25528) 2023-07-28 21:18:12 +02:00
api_notification_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_oauth2_apps_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_org_avatar_test.go Add API for changing Avatars (#25369) 2023-06-29 23:22:55 +00:00
api_org_test.go Revert package access change from #23879 (#25707) 2023-07-09 13:00:07 +00:00
api_packages_alpine_test.go Add v3.18 to TestPackageAlpine (#24972) 2023-05-29 15:45:32 +00:00
api_packages_cargo_test.go Fix profile render when the README.md size is larger than 1024 bytes (#25131) 2023-06-13 09:02:25 +00:00
api_packages_chef_test.go
api_packages_composer_test.go
api_packages_conan_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
api_packages_conda_test.go
api_packages_container_test.go Test if container blob is accessible before mounting (#22759) 2023-07-09 11:24:43 +00:00
api_packages_cran_test.go Add CRAN package registry (#22331) 2023-05-22 10:57:49 +08:00
api_packages_debian_test.go Fix handling of Debian files with trailing slash (#26087) 2023-07-24 16:19:44 +02:00
api_packages_generic_test.go Add direct serving of package content (#25543) 2023-07-03 15:33:28 +02:00
api_packages_goproxy_test.go Add Go package registry (#24687) 2023-05-14 23:38:40 +08:00
api_packages_helm_test.go
api_packages_maven_test.go
api_packages_npm_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_packages_nuget_test.go Fix handling of plenty Nuget package versions (#26075) 2023-07-26 19:43:21 +00:00
api_packages_pub_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
api_packages_pypi_test.go
api_packages_rpm_test.go Fix version in rpm repodata/primary.xml.gz (#26009) 2023-07-21 19:41:38 +00:00
api_packages_rubygems_test.go
api_packages_swift_test.go
api_packages_test.go Revert package access change from #23879 (#25707) 2023-07-09 13:00:07 +00:00
api_packages_vagrant_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_private_serv_test.go
api_pull_commits_test.go
api_pull_review_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
api_pull_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_releases_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_archive_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_avatar_test.go Add API for changing Avatars (#25369) 2023-06-29 23:22:55 +00:00
api_repo_branch_test.go Fix bug of branches API with tests (#25578) 2023-07-01 10:52:52 +08:00
api_repo_collaborator_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_edit_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_file_create_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
api_repo_file_delete_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_file_get_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_file_helpers.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
api_repo_file_update_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
api_repo_files_change_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
api_repo_get_contents_list_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_get_contents_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_git_blobs_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_git_commits_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_git_hook_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_git_notes_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_git_ref_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_git_tags_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_git_trees_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_hook_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_languages_test.go
api_repo_lfs_locks_test.go
api_repo_lfs_migrate_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_lfs_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_raw_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_tags_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_teams_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_repo_test.go Add commits dropdown in PR files view and allow commit by commit review (#25528) 2023-07-28 21:18:12 +02:00
api_repo_topic_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_settings_test.go
api_team_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_team_user_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_token_test.go Remove "misc" scope check from public API endpoints (#26134) 2023-07-26 02:53:31 +00:00
api_user_avatar_test.go Add API for changing Avatars (#25369) 2023-06-29 23:22:55 +00:00
api_user_email_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_user_follow_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_user_heatmap_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_user_info_test.go Fix API leaking Usermail if not logged in (#25097) 2023-07-31 01:18:38 +00:00
api_user_org_perm_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_user_orgs_test.go Allow Organisations to have a E-Mail (#25082) 2023-07-25 08:26:27 +00:00
api_user_search_test.go Fix API leaking Usermail if not logged in (#25097) 2023-07-31 01:18:38 +00:00
api_user_star_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_user_watch_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
api_wiki_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
attachment_test.go Use fetch to send requests to create issues/comments (#25258) 2023-06-16 06:32:43 +00:00
auth_ldap_test.go Fix the truncate and alignment problem for some admin tables (#26042) 2023-07-22 10:54:48 +00:00
avatar.png Add API for changing Avatars (#25369) 2023-06-29 23:22:55 +00:00
benchmarks_test.go
branches_test.go Improve AJAX link and modal confirm dialog (#25210) 2023-06-13 12:10:10 +00:00
change_default_branch_test.go
cmd_keys_test.go Refactor to use urfave/cli/v2 (#25959) 2023-07-21 17:28:19 +08:00
compare_test.go
cors_test.go
create_no_session_test.go Refactor web package and context package (#25298) 2023-06-18 09:59:09 +02:00
csrf_test.go
delete_user_test.go
download_test.go
dump_restore_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
editor_test.go
empty_repo_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
eventsource_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
explore_repos_test.go
git_clone_wiki_test.go
git_helper_for_declarative_test.go
git_smart_http_test.go
git_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
goget_test.go
gpg_git_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
html_helper.go
incoming_email_test.go
integration_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
issue_test.go Refactor and enhance issue indexer to support both searching, filtering and paging (#26012) 2023-07-31 06:28:53 +00:00
lfs_getobject_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
lfs_local_endpoint_test.go
lfs_view_test.go
links_test.go Serve pre-defined files in "public", add "security.txt", add CORS header for ".well-known" (#25974) 2023-07-21 12:14:20 +00:00
markup_external_test.go
migrate_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
mirror_pull_test.go
mirror_push_test.go
nonascii_branches_test.go
oauth_test.go Allow for PKCE flow without client secret + add docs (#25033) 2023-06-03 05:59:28 +02:00
org_count_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
org_project_test.go Fix access check for org-level project (#26182) 2023-07-29 14:13:24 +00:00
org_team_invite_test.go
org_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
private-testing.key
privateactivity_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
pull_compare_test.go
pull_create_test.go Make backend code respond correct JSON when creating PR (#25353) 2023-06-19 08:25:36 +00:00
pull_diff_test.go Add commits dropdown in PR files view and allow commit by commit review (#25528) 2023-07-28 21:18:12 +02:00
pull_merge_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
pull_review_test.go
pull_status_test.go Remove commit status running and warning to align GitHub (#25839) 2023-07-21 16:24:36 +08:00
pull_update_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
README_ZH.md replace drone exec to act_runner exec in test README.md (#24791) 2023-05-18 19:48:47 +00:00
README.md replace drone exec to act_runner exec in test README.md (#24791) 2023-05-18 19:48:47 +00:00
release_test.go
rename_branch_test.go Sync branches into databases (#22743) 2023-06-29 10:03:20 +00:00
repo_activity_test.go
repo_branch_test.go Newly pushed branches hints on repository home page (#25715) 2023-07-08 05:19:00 +02:00
repo_commits_search_test.go
repo_commits_test.go Remove commit status running and warning to align GitHub (#25839) 2023-07-21 16:24:36 +08:00
repo_fork_test.go
repo_generate_test.go Substitute variables in path names of template repos too (#25294) 2023-06-20 21:14:47 +00:00
repo_migrate_test.go
repo_search_test.go
repo_tag_test.go
repo_test.go Update octicons and use octicon-file-directory-symlink (#25453) 2023-06-22 22:05:52 +00:00
repo_topic_test.go
repo_watch_test.go
repofiles_change_test.go Refactor "Content" for file uploading (#25851) 2023-07-18 18:14:47 +00:00
setting_test.go
signin_test.go
signout_test.go Use JSON response for "user/logout" (#25522) 2023-06-26 21:36:10 +02:00
signup_test.go
ssh_key_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
timetracking_test.go
user_avatar_test.go
user_test.go Redesign Scoped Access Tokens (#24767) 2023-06-04 20:57:16 +02:00
version_test.go
view_test.go
webfinger_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
xss_test.go

Integration tests

Integration tests can be run with make commands for the appropriate backends, namely:

make test-sqlite
make test-pgsql
make test-mysql
make test-mysql8
make test-mssql

Make sure to perform a clean build before running tests:

make clean build

Run tests via local act_runner

Run all jobs

act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest

Warning: This file defines many jobs, so it will be resource-intensive and therefor not recommended.

Run single job

act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>

You can list all job names via:

act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l

Run sqlite integration tests

Start tests

make test-sqlite

Run MySQL integration tests

Setup a MySQL database inside docker

docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql

Run pgsql integration tests

Setup a pgsql database inside docker

docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql

Run mssql integration tests

Setup a mssql database inside docker

docker run -e "ACCEPT_EULA=Y" -e "MSSQL_PID=Standard" -e "SA_PASSWORD=MwantsaSecurePassword1" -p 1433:1433 --rm --name mssql microsoft/mssql-server-linux:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=gitea_test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql

Running individual tests

Example command to run GPG test:

For SQLite:

make test-sqlite#GPG

For other databases(replace mssql to mysql, mysql8 or pgsql):

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql#GPG

Setting timeouts for declaring long-tests and long-flushes

We appreciate that some testing machines may not be very powerful and the default timeouts for declaring a slow test or a slow clean-up flush may not be appropriate.

You can either:

  • Within the test ini file set the following section:
[integration-tests]
SLOW_TEST = 10s ; 10s is the default value
SLOW_FLUSH = 5S ; 5s is the default value
  • Set the following environment variables:
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite