Копия источника

https://arxiv.org/html/2512.09483v1

Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines Title: Content selection saved. Describe the issue below: Description: arXiv is now an independent nonprofit! Learn more × License: arXiv.org perpetual non-exclusive license arXiv:2512.09483v1 [cs.CL] 10 Dec 2025 LLM-SE LLM-based Search Engine LLM Large Language Model RAG Retrieval-Augmented Generation SEO Search Engine Optimization TSE Traditional Search Engine CV Coefficient of Variation MBFC Media Bias/Fact Check LLM-SE LLM-based Search Engine SD standard deviation JS Jaccard Similarity WD Wasserstein Distance RTD Rank Turbulence Divergence HTML HyperText Markup Language SHAP Shapley Additive exPlanation Source Coverage and Citation Bias in LLM-based vs.

Traditional Search Engines Peixian Zhang Note: Both authors contributed equally to this research. Affiliation: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China email: pzhang041@connect.hkust-gz.edu.cn , Qiming Ye Affiliation: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China email: qiming@connect.hkust-gz.edu.cn , Zifan Peng Affiliation: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China email: zpengao@connect.hkust-gz.edu.cn , Kiran Garimella Affiliation: Rutgers University, New Brunswick, United States email: kiran.garimella@rutgers.edu and Gareth Tyson Affiliation: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China email: gtyson@ust.hk © none Abstract. aise introduces a new paradigm for information seeking. Unlike tse (e.g., Google), these systems summarize results, often providing limited citation transparency.

The implications of this shift remain largely unexplored, yet raises key questions regarding trust and transparency. In this paper, we present a large-scale empirical study of aise, analyzing 55,936 queries and the corresponding search results across six aise and two tse. We confirm that aise cites domain resources with greater diversity than tse. Indeed, 37 %37\text{\,}\mathrm{\%} of domains are unique to aise.

However, certain risks still persist: aise do not outperform tse in credibility, political neutrality and safety metrics. Finally, to understand the selection criteria of aise, we perform a feature-based analysis to identify key factors influencing source choice. Our findings provide actionable insights for end users, website owners, and developers. 1.

Introduction Search engines are one of the most frequently used services on the web. They operate by crawling and indexing web content (Inc., ), then ranking results based on things like relevance (Wikipedia, ), authority (Wikipedia, ), reputation (Wikipedia, ) and experience (Journal, ). For years, users have relied heavily on these rankings to access information (Nielsen, 2016; Haim et al., 2018; Trielli and Diakopoulos, 2019). However, the way users access and interact with information is changing with the rapid development of llm (Naveed et al., 2025).

Indeed, by 2027, it is projected that 90M American adults will use AI for search (Statista, ). Here, retrieval in LLM-based Search Engines (LLM-SEs) involves an LLM agent conducting search activities on behalf of the user, then packaging up the results in a bespoke and easy-to-consume text summary. However, to date, we know little about how this shift in information access will impact society at-large. Technically, aise diverge from tse along two dimensions.

First, Coverage is supposedly enhanced through rag (rag) (Lewis et al., 2021). rag works by expanding queries into sub-queries (Zhou et al., 2022) and retrieving information from local knowledge database or online sources (Yao et al., 2022; Trivedi et al., 2023; Nakano et al., 2021; Menick et al., 2022), reducing the chance of omissions. In contrast, users of tse must typically perform those sub-queries themselves. Second, Reliability is expected to improve as llm filter and re-rank citations with the reasoning capabilities of llm (Lewis et al., 2020), allowing users to access key insights without inspecting multiple sources.

Although useful, this introduces multiple layers of indirection, potentially damaging transparency. For example, prior studies indicate that rag may produce unexpected bias (Avula et al., 2025; Dai et al., 2024), and that llm are prone to hallucinations (Huang et al., 2025) or even harmful content generation (Giorgi et al., 2025). Researchers have found that participants who use aise are unaware of errors in the responses (Spatharioti et al., 2025), and that more malicious sources appear in aise than tse (Luo et al., 2025). The existing solution to these challenges is for aise to include source citations (Gao et al., 2023), offering external URLs that evidence the assertions made in the response.

This allows users to follow such URLs, and check the correctness of the LLM summaries. However, to date, we lack a systematic evaluation of how aise use these citations, nor how they differ to the results offered by tse. Therefore, this paper explores the source citation of aise, as well as their ability to deliver information transparency and reliability compared to tse. We evaluate six different aise, benchmarking the results they return against to two major tse (Google and Bing).

To the best of our knowledge, this is the first study to perform a comprehensive evaluation across a wide range of aise, offering insights into their sourcing behavior, reliability, and biases. We explore the following research questions: • RQ1: What are the sources cited by aise, and how do they differ from those returned by tse? • RQ2: How is the quality of sources cited by aise, and how does it compare to tse? • RQ3: Which features influence the likelihood of a website being cited by aise compared to tse?

Our contributions are as follows: (1) We present the first large-scale measurement study comparing Google and Bing with six popular aise. We gather a dataset of 55,936 queries, covering 124,287 unique domains and 1,418,733 unique citation hyperlinks. We will make this available for research use. (2) Beyond returning fewer sources, aise diverge from tse in content selection: 37 %37\text{\,}\mathrm{\%} of domains in aise results are absent from tse outputs.

aise exhibit a less concentrated distribution and favor domains with lower user popularity. (3) Gemini returns frequent far-left citations, while Grok has the lowest partisan neutrality; both rely on limited sources that further limit overall reliability. Besides, aise do not outperform tse in cyber safety despite lower source volumes. (4) Feature-based analysis shows that domains favored by aise generally exhibit more structured, hierarchical HTML, easier-to-read text, lower domain popularity, and more outlinks to reputable sources compared to tse.

2. Background & Related Work tse tse (e.g., Google (Inc., ), Bing (Corporation, )) use crawlers to discover pages via links and sitemaps (Inc., ). Indexers then build inverted indexes (e.g., TF-IDF (Wikipedia, )) with term frequencies and positions. Finally, ranking combines link analysis (e.g., PageRank (Wikipedia, )) and quality criteria (e.g., E-E-A-T (Journal, )).

The final output for the user is presented as a ranked list of links. These ranking positions shape user impressions and direct traffic to the linked domains (Guan and Cutrell, 2007; Pan et al., 2007). Though accounting for over half of global web traffic (Inc., ), tse also raise concerns. For example, users may blindly trust top-ranked results or well-known domains, reinforcing rank bias (Wang et al., 2016) and authority-domain bias (Ieong et al., 2012).

The prevalence of highly similar information can further reinforce echo chambers (Diaz-Diaz et al., 2022), which accelerates political polarization and introduces additional bias (Poudel and Weninger, 2025). aise In contrast, aise adopt a fundamentally different approach by integrating retrieval with language generation through the rag paradigm (Lewis et al., 2021). Following the Least-to-Most Prompting framework (Zhou et al., 2022), they first interpret user intent and decompose complex queries into sub-queries. These sub-queries are then used to retrieve relevant content from either local knowledge bases or online sources (Yao et al., 2022; Trivedi et al., 2023).

For queries requiring recent or specialized information (e.g., news), aise additionally perform real-time web retrieval (Nakano et al., 2021; Menick et al., 2022). The retriever then ranks and selects the most relevant texts for downstream processing. The llm generator then interprets these documents, synthesizes the retrieved content (Lewis et al., 2020), and produces a concise natural-language summary with inline citations (Gao et al., 2023). However, LLMs themselves may exhibit biases.

For example, misinformation in retrieved documents can propagate into generated answers (Deng et al., 2025). Prior work also demonstrates that llm exhibit political bias when rating news sources (Yang and Menczer, 2025) and may generate malicious links (Luo et al., 2025). Additionally, results from Search Arena (Miroyan et al., 2025) show that users are strongly influenced by the number and type of citations, even when these attributions are incorrect. Comparison of tse vs.

aise By shifting from link ranking to answer generation, aise mark a fundamental departure from the click-driven exploration of traditional search. Prior work shows that while aise can outperform tse on complex, decision-oriented tasks (Spatharioti et al., 2025), users also exhibit over-reliance on incorrect outputs (Kabir et al., 2023). In contrast, (Wazzan et al., 2024) demonstrate that tse users achieve higher accuracy in an image localization task, highlighting task-dependence in performance. Citations have been shown to increase user trust in aise responses (Ding et al., 2025), and topic analyses indicate that users prefer aise for semantically rich, language-intensive tasks (Caramancion, 2024).

However, aise also exhibit higher rates of unsafe or malicious link generation compared to tse (Luo et al., 2025). Together, these findings illuminate how users interact with aise. Yet, they do not study the sources that aise cite, nor how they differ from tse. Thus, in this work, we address this gap by conducting the first large-scale analysis of aise citation patterns.

We simulate the search activities on both tse and aise, focusing on their sourcing patterns and the implications for information exposure. 3. Data Collection and Annotation Figure 1. The Pipeline of Data Collection We begin by introducing our data collection and annotation methodologies.

Figure 1 presents the overall pipeline of our data collection work. Note that aise may process multiple pages before presenting the selected sources to the user. However, we focus exclusively on user-facing citations to capture the end-user experience. We capture a snapshot of popular queries to compare the sources returned by search engines.

3.1. Search Engines We define an aise as an LLM system that can retrieve web content and provide cited answers paired with summarized text. We therefore first identify publicly available llm that provide web search functionality. We include 6 widely used platforms: ChatGPT, Gemini, Perplexity, Grok, Google AI Mode (short as AI Mode), and Copilot Search in Bing (short as Copilot).

For comparison, we also include Google and Bing as representatives for tse. 3.2. Search Query Corpus Prior work (Miroyan et al., 2025; Yang et al., 2018) shows that tse favor keyword-based queries, whereas users express real-world factual intents in natural language on aise. Therefore, we augment the query corpus with keywords, related questions, and controversial queries.

Keyword Queries. We start by collecting a seed keyword dataset to represent search activities. Here, we use Google Trends and X (formerly Twitter) (Step 1(a) in Figure 1). These trending keywords serve as representative inputs, reflecting what users actively search for across social media and search engines.

Both of them are representative of real-time user interests and are widely used in the prior research (Annamoradnejad and Habibi, 2019; Jun et al., 2018). We download Google Trends11 1 https://trends.google.com/trends/ and X Trends by trends2422 2 https://trends24.in/united-states/ everyday between Jul. 16 to Aug. 10 2025 across the USA.

In total, we collect 7,519 unique keywords from Google Trends and 5,593 from X, and we include the search results corresponding to these keywords in our dataset. Related Question Query. Relying solely on single keywords is insufficient, as they may be ambiguous. Prior studies have shown that users interacting with aise tend to prefer sentence-style queries rather than isolated keywords (Caramancion, 2024; Luo et al., 2025; Wazzan et al., 2024).

To align with this behavior, we generate natural sentence queries from the above trending keyword lists. As illustrated in Figure 1 Step 1(b), we take the keywords generated from Step 1(a), and search for them using both Google Search and Google AI mode. As well as returning the search results, these two services also return a set of recommended related search queries. These are generated from prior real searches, and give insight into typical queries launched by users.

This approach ensures that our query set reflects realistic user input patterns. In total, this process yields 37,931 unique questions from Google search result and 2,612 unique questions from AI mode. Controversial Questions. To complement the related questions, we include an existing controversial query dataset to examine how search engines handle sensitive or polarizing content (Step 1(c) in Figure 1).

We utilize an open-source dataset containing sentence-level queries sourced from popular Quora questions that have been identified as controversial by large language models (LLMs) (Sun et al., 2023). This dataset covers 2,281 queries. These queries are curated to reflect socially and politically sensitive topics, enabling us to assess how different aise handle nuanced or polarizing content. 3.3.

Search Query and Response Processing Upon completing the construction of the search query dataset, we initiate the search process as Step 2 by inputting all keywords and sentence queries to both the tse and aise. We then employ DrissionPage33 3 a web automation tool based on Python (DrissionPage, ) to simulate user interactions with browser. Note, URLs may appear multiple times within a single response; however, we count each only once. tse.

To minimize personalization effects from tse (Hannak et al., 2013), we configure the browser to operate in incognito mode. Previous work shows that the click-through rate for the second page is around 0.63 %0.63\text{\,}\% on Google (Dean, 2025), thus we focus on first-page results to reflect the content most end users encounter (Gezici et al., 2021). We explicitly flag URLs marked with “Ad” or “Sponsored” tags as advertisements, while preserving their original positions in the ranked list for completeness. In total, our dataset contains 481,565 unique URLs from Google and 218,122 unique URLs from Bing.

aise. For aise, we collect data based on browser-based search mode.44 4 Note, some aise (e.g., ChatGPT and Gemini) provide the desktop version. We operate the browser in incognito mode to minimize potential personalization effects. Table 1 details the additional actions we performed for each aise.

Notably, ChatGPT is the only aise that mandates user authentication; thus, we configure it to avoid referencing prior interactions and disable memory features to ensure consistency across sessions. We then parse the full response payloads and extract the returned items and citations as the cited sources, as illustrated in Step 2 of Figure 1. Table 1. The Summary of AI Search Engines.

Name Version Login Search Button # of Cited Sources AI Mode Gemini-based* ✗ ✗ 414,524 ChatGPT gpt-5.0-instant ✓ ✓ 206,590 Copilot GPT-based* ✗ ✗ 1,029,015 Gemini Gemini-2.5-flash ✗ ✗ 182,541 Grok Grok 3.0 ✗ ✗ 62,420 Perplexity Sonar ✗ ✓ 280,699 AI Mode and Copilot do not publish exact version numbers. For clarity, our analysis exclusively uses Copilot in Bing. 3.4. Data Annotations Search Query Annotation.

We annotate each query according to Google Trends’ categorization scheme (see Appendix I). To this end, we employ a large language model (LLM) as a classifier. The detailed methodology and validation used for category prediction are provided in Appendix I. Overall, the query topics are broadly distributed across 26 categories.

The most represented categories are Arts & Entertainment (27 %27\text{\,}\mathrm{\%}), People & Society (19 %19\text{\,}\mathrm{\%}), and Sports (17 %17\text{\,}\mathrm{\%}), confirming strong topical diversity. Cited Domains. To categorize the cited websites, we employ the Advanced Classification Engine (ACE) from Forcepoint (ThreatSeeker, ). ACE performs URL categorization and is used in prior studies (Vallina et al., 2020).

To evaluate source popularity, we employ three widely used third-party datasets. To quantify the number of visits a website receives, we use the Tranco Ranking (Le Pochat et al., 2019b). To assess source quality, we incorporate two widely used third-party datasets: Media Bias/Fact Check (MBFC) (Zandt, ) for credibility and political bias, and VirusTotal (Sistemas, ) for security and malicious-domain detection. Together, these tools provide domain-level annotations on categorization (Bouwman et al., 2022), popularity (Le Pochat et al., 2019a), political bias (Shah et al., 2025), and cyber security risks (Peng et al., 2019).

Note, not all domains cited by the search engines can be matched to the above datasets; for example, MBFC only covers news websites. The detailed coverage rate and validation results are provided in Appendix C. 4. Source Citation Behaviors (RQ1) Unlike tse, which typically return a ranked list of hyperlinks, aise provide a summarized response supported by fewer selected URL sources.

However, it is unclear whether aise differ the information landscape with well-selected sources. To address this gap, we measure the sourcing behaviors beginning. 4.1. Comparing Citation Patterns We begin by analyzing the distribution of sources cited in the search results.

This allows us to assess how different engines prioritize, select, and present reference sources. (a) (b) Figure 2. ECDFs: number of unique (a) websites and (b) domains per response across search engines. Number of Sources.

We first assess the number of distinct sources embedded per response. Figure 2(a) and Figure 2(b) illustrate the distributions of unique URLs and domains.55 5 URLs or domains may appear multiple times within different parts of the response text due to repeated embedding by the search engines. As expected, all aise source fewer URLs (mean: 4.3) and domains (mean: 3.4) compared to tse (mean: 10.3 URLs and 7.3 domains). Across all aise, fewer than ten distinct URLs appear in 80% of responses.

Notably, Grok and Gemini are the least likely to cite external sources, with 82% and 38% of their responses containing no cited websites, respectively. This behavior may stem from a stronger reliance on internal knowledge bases or stricter display thresholds, causing these engines to prioritize model-generated content over external citations. However, prior work shows that limited sourcing can undermine user trust in search systems (Miroyan et al., 2025). Figure 3. Lorenz curve illustrating domain frequency distribution and corresponding Gini index.

Domain Concentration. Source concentration is a central concern in prior studies of tse (Diaz-Diaz et al., 2022), and similar risks have been identified in LLMs, where authority bias shapes how models evaluate content (Chen et al., 2024). We therefore use the Gini index to quantify domain concentration for each search engine, capturing the extent to which they disproportionately return certain domains. A higher Gini index (→1\to 1) indicates that a few domains dominate the citations, whereas a lower value (→0\to 0) suggests an even distribution.

Figure 3 presents the Lorenz Curve (Gastwirth, 1971) for each search engine. The results show that all aise exhibit lower Gini indices than tse, indicating that aise return a slightly more uniform spread of domains. To validate this, we perform a statistical test for each system pair (i.e. aise vs.

tse) following the methodology of (Xu, 2000) (details in Appendix B). Our results show that Gemini, Grok and ChatGPT differ significantly from more than one tse. ChatGPT diverges only from Google (p=0.026p=0.026), likely reflecting its reliance on Bing, whereas Gemini and Grok differ significantly from both Bing and Google (p<0.001p<0.001). Grok reports using no third-party provider, while Gemini depends on Google, and we conjecture that their divergence may also stem from their lower citation counts (see §2).

Although Gemini and Grok exhibit lower Gini indices than the tse, they nonetheless return fewer sources overall. However, the extent to which such source diversity impacts the actual user retrieval experience remains an open question. This motivate us to measure the source quality in §5. Figure 4.

Percentage of domain overlap: Unique indicates domains appearing in only one search engine type, while Common indicates sharing between aise and tse. 4.2. Disparity in Domain Selection We next assess the extent to which each search engine returns unique domains. High distinctiveness indicates that users may retrieve varied information depending on the search engine.

Overall Disparity. Figure 4 presents the fraction of domains that are unique to aise, unique to tse, or overlapping between the two. Here, Unique refers to domains appearing exclusively in the corresponding aise or tse, whereas Common denotes domains shared between both. Surprisingly, large differences are observed between aise and tse: only 38% of domains appear in both (i.e.

common), whereas 37% are unique to aise results. Similar patterns also persist at the response level, with an average overlap of less than 40% between any aise and tse pair (see Appendix D). Even for services provided by the same company (e.g., Google, AI Mode, and Gemini), the sourced domains differ substantially (≥30%\geq 30\%). This divergence likely arises because the aise decomposes the original queries rather than using the full user-provided context when retrieving content from external sources (Press et al., 2023).

This confirms that users relying solely on aise are exposed to a distinct set of information sources compared to those using tse alone. (a) With Google (b) With Bing Figure 5. RTD of news relevant domains distribution between aise and (a) Google (b) Bing. The domains on the left are more likely to appear in aise response and domains on the right are more likely to appear in tse response.

Disparity in News Sources. Although aise diverge from tse in their overall domain distribution, it remains unclear whether they favor specific sources. Such preferences are important, as over-reliance on a single source can amplify biases and reduce diversity, a concern particularly relevant for news content. To investigate this, we extract all news domains (as labeled by VirusTotal, see §3.4) from the search results.

This results in 2,675 unique domains returned by both aise and tse. We then employ Rank Turbulence Divergence (RTD) (Dodds et al., 2023; Poudel and Weninger, 2024) to quantify differences between aise and tse results (see Appendix E for details). RTD measures domain-specific preference: a score close to 11 indicates strong presence in tse results, whereas a score near −1-1 signals a higher likelihood of appearing in aise. Figure 5 presents the rtd (rtd) distributions across domains for aise compared with (a) Google and (b) Bing.

Each subplot highlights the top-ranked domains with the highest rtd values cited by the respective engines. Although the overall difference centers near zero, certain domains receive disproportionate emphasis. As an illustrative case, ChatGPT more frequently surfaces the original news agencies Reuters and AP News (Google RTD = –0.48, Bing RTD = –0.41). In contrast, tse favor established broadcasters (CNN, USA TODAY) and opinion-based sources (Townhall).

Since news sources often have established political positions, such divergence could introduce bias, which we further investigate in §5.1. 4.3. Measuring Source Popularity Finally, the popularity of domains is measured through two metrics, as it can directly influence the perceived authority of a search engine: (1) ranking positions on Google and Bing, where higher placements indicate greater search-driven exposure (Inc., );66 6 Note that organic search via tse still accounts for over half of global web traffic as of 2025 (Inc., ). (2) Tranco rankings (Pochat et al., 2018), providing a platform-agnostic estimate of web traffic for each domain.

Together, these metrics capture both the likelihood of domains being discovered and visited, serving as a proxy for domain popularity and its impact on perceived search authority. Figure 6. The figure shows where overlapping domains between aise and tse appear in each search. It aggregates across all searches to report the frequency of these domains’ ranks in tse.

Search-driven Exposure. We begin by assessing the rank positions of the domains that appear in both the aise responses and the top tse results. For each search query, we identify all domains that overlap between aise and tse in a pairwise comparison. This accounts for only 37% of domains in our dataset (see §4.1).

If a domain appears at multiple ranks within tse results for a single search, all instances are counted to reflect repeated visibility. We then extract the overlapping domains for each aise query along with their corresponding rank positions in the tse.77 7 Sponsored content and advertisements are excluded to focus solely on organic traffic. Finally, we calculate the frequency of each rank position across all search results for each aise. Given that user attention concentrates disproportionately on the top search results (Inc., ), this ranking position as a indicator of search-driven exposure.

Figure 6 illustrates the rank distribution of overlapping domains in the tse results. Surprisingly, we observe heavy skew in overlapping domains toward the top-1 tse results. The sources aise most frequently cite are those ranked first by traditional search engines 23.27% in Bing and 14.53% in Google. This suggests that for the overlapped search results, aise tends to cite the domains that are also considered as the most important in tse. Traffic-Based Popularity.

We then use the Tranco ranking (Pochat et al., 2018) to assess the user popularity of the cited domains. Tranco has been widely adopted in prior work (Bilot et al., 2024; Trautwein et al., 2022; Kassem et al., 2022; Gunawan et al., 2021) and allows us to assess whether aise prioritize high-traffic websites or rely on less popular sources. Here, a lower rank indicates higher popularity. Table 2 reports the mean and median Tranco ranks for all domains cited by each search engine.

A Kruskal-Wallis test (McKight and Najab, 2010) is further conducted to confirm statistical differences (H=58627.99H=58627.99, p<0.001p<0.001). Post-hoc results show all aise differ significantly from tse (p<0.001p<0.001). Overall, except for ChatGPT, the aise tend to return less popular domains (i.e. higher rank) than the tse.

Notably, Copilot and Gemini exhibit substantially higher average Tranco ranks with over 22,000 above both tse, which indicates a stronger inclination toward low-traffic domains. Such a tendency may negatively affect user trust, as search behavior research (Pan et al., 2007) shows that users typically perceive high-traffic domains as more authoritative. Beside, lower-traffic domains may also be less reliable, which motivates our investigation of domain credibility in §5.1. Table 2.

Mean and median Tranco ranks across search engines, emphasizing the divergence Δ\Delta of aise from the tse baseline. Search Engine Mean(μ\mu) Median Δ\Delta vs Google (μ\mu) Δ\Delta vs Bing (μ\mu) Bing 48124.95 2412.0 +670.16 0.00 Google 41427.79 746.0 0.00 -697.16 AIMode 43118.93 1809.0 +691.14 -1006.02 ChatGPT 40442.48 713.0 -985.31 -1682.47 Copilot 69054.10 3912.0 +27626.31 +20929.15 Gemini 64121.03 3805.0 +22693.24 +15996.08 Grok 42846.77 2507.0 +418.98 -1278.18 Perplexity 48208.00 1362.0 +6780.21 +83.05 5. Source Quality (RQ2) In the previous section, we showed that aise return a more diverse set of domains than tse. However, the shift in retrieval format alters the user’s role: instead of browsing multiple links, users receive an llm-generated summary with only a few embedded references.

This places greater emphasis on the quality of those few sources. Accordingly, RQ2 evaluates source quality through two axis: reliability and cyber threats. (a) (b) Figure 7. Histograms of (a) credibility scores and (b) political scores for all domains based on mbfc.

5.1. Reliability We next study if aise cite more reliable sources, as compared to tse. We focus primarily on news and media landscapes, as these sources are most directly linked to information bias and shape users’ perceptions of factual accuracy (Li and Aral, 2025). Assigning Scores To evaluate this, we use the ratings from mbfc (Zandt, ) to assess political leaning (i.e.

ownership and editor affiliation) and credibility (i.e. transparency and accuracy) of cited domains.88 8 Note, domains without mbfc records are excluded. Detailed matching rates are shown in §3. 51 %51\text{\,}\mathrm{\%} of ratings provide only categorical labels rather than the full scores.

Following the official methodology (Zandt, ), we therefore assign the average value of each category for the missing scores. Note that the original credibility scores range from −1-1 to 1010. To align with the political leaning scale (−10-10 to 1010) and facilitate visualization (with 00 as the neutral midpoint), we transform credibility scores to range from −10-10 (least factual) to 1010 (most factual) (see Appendix §F). Both the credibility and political scores are statistically significant according to the Kruskal–Wallis test (see Appendix G).

Such difference indicates that the distinct sourcing practices described in §4.2 indeed affect overall source quality. Credibility. We first examine the credibility of the response using the factual-reporting ratings from mbfc (Zandt, ). Figure 7 presents the credibility scores for all domains returned across search engines.

The prominent spike visible in all distributions is driven largely by Wikipedia (score: −0.9-0.9), which represents nearly one-third (32 %32\text{\,}\mathrm{\%}) of all domain occurrences. Bing outperform other search engines, primarily due to its pronounced concentrations around around bing.com (score: 6.36.3) and cambridge.org (score: 8.38.3), comprising 28 %28\text{\,}\mathrm{\%} and 9 %9\text{\,}\mathrm{\%} of its cited domains, respectively. As for aise, except for Gemini and ChatGPT, other aise fall between Bing and Google. However, such concentration of domains in response level also brings troubles.

For example, Gemini shows an exceptionally high reliance on Wikipedia (score: −0.9-0.9), which accounts for 41 %41\text{\,}\mathrm{\%} of its citations, and 62 %62\text{\,}\mathrm{\%} of its responses cite only a single mbfc-listed domain. This narrow sourcing not only reduces variance but also corresponds to its lowest credibility scores (in both mean and median). These results suggest that greater source diversity is an important factor for improving credibility. Political Leaning.

The political leaning of domains is rated by mbfc (Zandt, ).99 9 Note, the score is termed “Bias” by mbfc. Figure 7 illustrates that the political scores for all domains returned by the search engines range from -10 (left) to +10 (right). Overall, given our set of queries, all engines display a left-leaning political orientation, with both mean and median scores below zero. Except for Grok, the mean is lower than the median for other search engines, which confirms a set of extreme left-leaning outliers.

This pattern reinforces the overall left-leaning tendency across engines. For example, 6.4 %6.4\text{\,}\mathrm{\%} of domains cited by Gemini fall within the far- or extreme-left range (<−5<-5). This bias is exacerbated by Gemini’s tendency to cite fewer sources than comparable systems (see §4.1), thereby concentrating user exposure on a narrow set of ideologically skewed domains. To capture the aggregate bias presented to users, we further evaluate political neutrality at the response level by computing the mean political score of all cited domains.

Under this metric, Grok exhibits the lowest neutrality with only 16 %16\text{\,}\mathrm{\%} of responses achieving a score of 00, contradicting public claims of balanced positioning.1010 10 https://www.nytimes.com/2025/09/02/technology/elon-musk-grok-conservative-chatbot.html Ultimately, the reliance of both Gemini and Grok on the fewest distinct domains underscores the critical role of source diversity in mitigating political bias. 5.2. Cyber Threat Finally, we assess the cyber threats to ensure that source credibility does not compromise user safety. For this, each domain is checked using VirusTotal,1111 11 https://www.virustotal.com which aggregates reports from 97 security vendors (Choo et al., 2024).

To balance false positives and false negatives, we adopt a threshold of two or more positive flags to classify a domain as unsafe, following prior work (Peng et al., 2019). Alternative thresholds are evaluated in Appendix H. This approach allows us to compare whether aise are more likely than tse to include unsafe sources. Exposure Risk Among 125,555 unique domains returned by search engines, 439 (0.36 %0.36\text{\,}\mathrm{\%}) are labeled as malicious.

The distribution follows a log-scale dominated by a small subset of domains. The top 10 %10\text{\,}\mathrm{\%} of malicious domains account for 59 %59\text{\,}\mathrm{\%} of all occurrences, with the most frequent domain appearing 137 times. Within this dominant subset, Google (19 %19\text{\,}\mathrm{\%}) and Copilot (18 %18\text{\,}\mathrm{\%}) contributes the largest share of total malicious domains returned. Table 3 provides a further assessment of exposure risks (i.e.

the likelihood of encountering malicious domains) across search engines. As baselines, Bing and Google tse generate 0.42 %0.42\text{\,}\mathrm{\%} and 0.23 %0.23\text{\,}\mathrm{\%} malicious domains among all cited domains, respectively. Most aise exhibit risks that fall between these two baselines, despite sourcing fewer number of domains per response. We also assess the severity of maliciousness by measuring the average number of malicious domains per response.

Only Gemini show slightly higher averages compared to both of the baselines. Although Gemini and Google operate within the same ecosystem, Gemini yields a higher mean (Avg = 1.56, ×\times1.5 of Google). These findings suggest that while aise may cite fewer sources overall, they do not necessarily mitigate exposure to risky domains. Moreover, engines within the same ecosystem can behave inconsistently in threat sourcing.

This motivates us to further examine: (1) whether overlapping sets of malicious domains appear across engines for the same query; and (2) which types of queries are more likely to trigger such risks. Table 3. Cyber Threat Exposure Risk Across Search Engines Metric AI Mode ChatGPT Copilot Gemini Grok Perplexity Bing T. Google T.

% of Threat Domains 0.25 %0.25\text{\,}\mathrm{\%} 0.32 %0.32\text{\,}\mathrm{\%} 0.34 %0.34\text{\,}\mathrm{\%} 0.26 %0.26\text{\,}\mathrm{\%} 0.34 %0.34\text{\,}\mathrm{\%} 0.23 %0.23\text{\,}\mathrm{\%} 0.42 %0.42\text{\,}\mathrm{\%} 0.23 %0.23\text{\,}\mathrm{\%} Avg. of Threat Domains per Resp. 1.07 1.01 1.03 1.56 1.11 1.02 1.33 1.05 Numbers that exceed either of the baselines from Google or Bing (tse) are displayed in bold, while those exceeding both baselines are shown in italic. Overlapping Domains.

We use the Jaccard Similarity to measure the similarity of domains between malicious responses generated for the same query across different search engines. A Jaccard similarity above 0.5 is considered high and indicates substantial overlap between two sets (Rajaraman and Ullman, 2011). This analysis examines whether engines tend to return overlapping malicious sources. Overall, 85 %85\text{\,}\mathrm{\%} of engine pairs share an average score above 0.5, and 14 %14\text{\,}\mathrm{\%} exceed 0.8, indicating that the malicious domain sets are largely identical across engines for the same query.

This high overlap in malicious domains implies that coordinated mitigation strategies targeting shared malicious sources could reduce the spread of harmful content across search engine optimization. Search Query Category. We finally compute the Jaccard similarity for query sets that trigger malicious domains to assess whether specific types of queries are more prone to such threats. All Jaccard values fall below 0.4, indicating that malicious domains are triggered by distinct queries.

However, potential overlaps in categories may still exist. To explore this, we employ an llm to map each query to its corresponding category (details in Appendix I). Over 50 %50\text{\,}\mathrm{\%} of queries that return malicious domains fall into four primary categories: arts & entertainment, people & society, finance, and sports (see Appendix K). This indicates that aise may return malicious domains even for seemingly innocuous topics.

6. aise vs. tse Website Characteristics From the previous analysis, we confirm that aise represents a new search paradigm that diverges from tse ranking mechanisms. To better understand the mechanisms underlying aise citation behavior, we conduct a feature-based analysis to identify key factors that influence source selection in aise.

6.1. Features in HTML We begin by analyzing key html (html)-based features. Prior studies (Tan et al., 2025; Lastowka, 2000; Thurow, 2003) show that HTML structure strongly influences how both aise and tse retrieve and cite content. This influence arises from two aspects: the plain text and the underlying structure of HTML (Tan et al., 2025).

Accordingly, we analyze both types of HTML features. We randomly sample 10,000 URLs from each comparison set.1212 12 Note that all selected domains explicitly permit data crawling, as verified via their robots.txt. For the plain text, we assess the readability of the content and compute two widely used readability metrics: Flesch Reading Ease Score (Kincaid et al., 1975) and Flesch–Kincaid Grade Level (Flesch, 2007). For the html structure, we extract four features: semantic tags, DOM nesting depth, accessibility attributes, and counts of deprecated or discouraged tags.

These metrics are detailed in Appendix L. Table 4. Kolmogorov-Smirnov (KS) Test Results for Feature Distributions Between Groups. Feature KS p-value Mean Semantic Tags 0.061 p<0.01p<0.01 aise>tse Nesting Depth 0.069 p<0.01p<0.01 aise>tse Accessibility Features 0.066 p<0.01p<0.01 aise=tse Markup Errors 0.022 p<0.01p<0.01 aise=tse Flesch Reading Ease Score 0.096 p<0.01p<0.01 aise>tse Flesch–Kincaid Grade Level 0.082 p<0.01p<0.01 aise<tse To assess the distributional differences between aise and tse, we use the Kolmogorov–Smirnov (KS) test to compare their empirical distributions.

Table 4 presents the KS statistics and mean comparisons for the six features. All differences are statistically significant (p<0.01p<0.01) across both textual readability and structural characteristics. In terms of readability, tse content is considerably more complex, with an average grade level of 18.24 (vs. 14.57 for aise) and a reading ease score of 12.32 (vs.

24.15). These results suggest that aise tend to favor more accessible, less textually demanding content when selecting sources. For structural features, aise pages show a modest increase in HTML tag usage (mean: 7 vs. 6) and nesting depth (mean: 16 vs.

15) compared to tse pages. The nesting depth feature measures the maximum level of HTML tag embedding within a webpage, reflecting the structural complexity of its DOM hierarchy. Overall, aise appear to prioritize sources that are both more readable and structurally informative, reflecting a preference for content that is easier for both humans and machines to process. 6.2.

Predicting Unique Inclusion of Domains Beyond HTML features, our analyses in §5 indicate that domain-level annotation features may also influence sourcing preferences. To quantify their impact, we train predictive models and examine feature importance, identifying the factors that most strongly differentiate domains that are uniquely cited by aise from those cited by tse. Table 5. Overview of input features and preprocessing steps used in the classification pipeline.

Feature Type Feature Name(s) Transformation Numeric Tranco ranking, the number of out-links from VT MBFC bias and factual Mean imputation Standard Scaler Categorical domain, media category from VT Most frequent imputation One-Hot Encoding Textual search query, description from VT TF-IDF Classifier Design. We train a classifier to differentiate domains that uniquely appear in aise from those that are exclusive to tse, aiming to identify the characteristics that distinguish their respective source selections. The input features are structured into three modalities: numeric, categorical, and textual shows in Table 5. We perform a stratified 80:20 train–test split, and select models using 5-fold stratified cross-validation on the training set.

Five classifier types are evaluated, with hyperparameter tuning conducted via grid search to optimize performance. Detailed hyperparameter configurations are provided in Appendix J. Model Evaluation. Table 6 reports per-class F1 scores for cited and non-cited labels, along with the overall weighted F1 score.

The best-performing model is XGBoost, achieving a weighted F1 score of 0.758. The model performs notably better in identifying domains unique to aise, with a precision of 0.668 and a high recall of 0.879, suggesting strong sensitivity in detecting such domains. Table 6. Per-class and overall weighted F1 scores for predicting whether a domain is cited exclusively by aise.

LR RF XGBoost KNN MLP AI-SE 0.693 0.740 0.759 0.697 0.708 T-SE 0.576 0.505 0.450 0.439 0.519 F1 0.692 0.727 0.758 0.697 0.707 Feature Importance. Finally, we analyze the feature importance of the best-performing model, XGBoost, using the SHAP framework (Lundberg and Lee, 2017). Six features have mean SHAP values exceeding 0.1. The most influential are the Tranco ranking (0.923), number of outlinks (0.799), and the .com top-level domain (0.623), followed by subdomain count (0.189), .nl domains (0.135), and the business and economy category (0.116).

These features collectively exert a strong positive influence on the model output. The Tranco ranking captures a domain’s global popularity based on aggregated web traffic rankings. Note, a higher numerical rank corresponds to lower popularity comparing. This aligns with our findings in Section 5 aise favor for less popular domains comparing with tse.

The number of outlinks represents how many external references a webpage contains; a larger number of outlinks may reflect richer contextualization and stronger integration with other resources. 7. Conclusion and Future work Conclusion. We present the first large-scale empirical study of the sourcing behaviors of aise, covering 55,936 question across six aise and two tse.

§4 reveals that aise diverge from the sourcing paradigm of tse. aise return far fewer URLs per response (4.34.3 vs. 1010 on average), yet 37 %37\text{\,}\mathrm{\%} of the domains are absent from tse. §5 further assesses the sourcing quality of this novel search behavior across two dimensions: reliability and cyber threats.

We find that aise return less popular domains, on average. Thus, overall reliability depends heavily on these selections, with source diversity serving as a key indicator of credibility. Moreover, despite returning far fewer sources, aise exhibit risk levels comparable to those of tse. Finally, §6 identifies the features influencing source selections: aise are biased toward established, well-referenced information environments.

This potentially improving citation quality yet narrowing the diversity of accessible content. Together, these findings highlight the need for more diverse and safer sourcing practices in future aise. Future Work. The architecture of aise is multifaceted, encompassing document retrieval, internal ranking, summarization, and citation selection.

Our analysis restricts its scope to user-facing citations to best assess the impact on end-users, yet lacking the measuring the intermediate retrieval pipeline effectively. Concurrently, commercial imperatives (e.g., sponsored content) raise concerns regarding the influence of financial incentives on information prioritization. We identify two critical avenues for future research. First, future work could audit full retrieval traces to reveal potential biases in pre-generation filtering and ranking.

For example, we notice that some aise (e.g., Grok) disclose their internal reasoning and retrieval steps recently, which offer us a new opportunity to examine the factors underlying their pre-generation process. Second, research could examine monetization-driven behaviors, such as Generative Engine Optimization (GEO), to understand how commercial incentives influence content delivery. Notably, ChatGPT launched a shopping assistant in their recent product.1313 13 https://openai.com/index/chatgpt-shopping-research/ GEO providers can boost the visibility of certain brands in the outputs by hijacking specific keywords, thereby influencing the final source selection. Exploring these directions will clarify the impact of aise on transparency and information integrity.

References Alibaba Cloud Qwen Team (2025) Alibaba Cloud Qwen Team Qwen3: think deeper, act faster. Note: https://qwenlm.github.io/blog/qwen3/Accessed: 2025-11-27. Release of Qwen 3 dense models, including 32B variant. Cited by: Table 10, Table 10.

Annamoradnejad and Habibi (2019) I. Annamoradnejad and J. Habibi A comprehensive analysis of twitter trending topics. In 2019 5th international conference on web research (ICWR), pp.

22–27. Cited by: §3.2. Avula et al. (2025) S.

Avula, C. Lee, R. Zhang, and V. Murdock Measuring the fairness gap between retrieval and generation in rag systems using a cognitive complexity framework.

In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’25, New York, NY, USA, pp. 2994–2998. External Links: ISBN 9798400715921, Link, Document Cited by: §1. Bilot et al.

(2024) T. Bilot, N. El Madhoun, K. Al Agha, and A.

Zouaoui A survey on malware detection with graph representation learning. ACM Computing Surveys 56 (11), pp. 1–36. Cited by: §4.3.

Bouwman et al. (2022) X. Bouwman, V. Le Pochat, P.

Foremski, T. Van Goethem, C. H. Gañán, G.

C. Moura, S. Tajalizadehkhoob, W. Joosen, and M.

Van Eeten Helping hands: measuring the impact of a large threat intelligence sharing community. In 31st USENIX Security Symposium (USENIX Security 22), pp. 1149–1165. Cited by: §3.4.

Caramancion (2024) K. M. Caramancion Large language models vs. search engines: evaluating user preferences across varied information retrieval scenarios.

arXiv preprint arXiv:2401.05761. Cited by: §2, §3.2. Chen et al. (2024) G.

H. Chen, S. Chen, Z. Liu, F.

Jiang, and B. Wang Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669.

Cited by: §4.1. Choo et al. (2024) E. Choo, M.

Nabeel, D. Kim, R. De Silva, T. Yu, and I.

Khalil A large scale study and classification of virustotal reports on phishing and malware urls. SIGMETRICS Perform. Eval. Rev.

52 (1), pp. 55–56. External Links: ISSN 0163-5999, Link, Document Cited by: Appendix H, §5.2. [9] M.

Corporation Bing search development document. Note: https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a Cited by: §2. Dai et al. (2024) S.

Dai, C. Xu, S. Xu, L. Pang, Z.

Dong, and J. Xu Bias and unfairness in information retrieval systems: new challenges in the llm era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6437–6447.

Cited by: §1. Dean (2025) B. Dean We analyzed 4 million google search results. here’s what we learned about organic click through rate.

Note: https://backlinko.com/google-ctr-statsBacklinko Cited by: §3.3. DeepSeek AI (2024) DeepSeek AI DeepSeek-v3 technical report. Note: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdfAccessed: 2025-11-27. Details 671B MoE model with 37B active parameters.

Cited by: Table 10. Deng et al. (2025) B. Deng, W.

Wang, F. Zhu, Q. Wang, and F. Feng Cram: credibility-aware attention modification in llms for combating misinformation in rag.

In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, pp. 23760–23768. Cited by: §2.

Diaz-Diaz et al. (2022) F. Diaz-Diaz, M. San Miguel, and S.

Meloni Echo chambers and information transmission biases in homophilic and heterophilic networks. Scientific Reports 12 (1), pp. 9350. Cited by: §2, §4.1.

Ding et al. (2025) Y. Ding, M. Facciani, E.

Joyce, A. Poudel, S. Bhattacharya, B. Veeramani, S.

Aguinaga, and T. Weninger Citations and trust in llm generated responses. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, pp.

23787–23795. Cited by: §2. Dittrich and Kenneally (2012) D. Dittrich and E.

Kenneally The Menlo Report: Ethical Principles Guiding Information and Communication Technology Research. Technical report U.S. Department of Homeland Security. External Links: Document Cited by: Appendix A.

Dodds et al. (2023) P. S. Dodds, J.

R. Minot, M. V. Arnold, T.

Alshaabi, J. L. Adams, D. R.

Dewhurst, T. J. Gray, M. R.

Frank, A. J. Reagan, and C. M.

Danforth Allotaxonometry and rank-turbulence divergence: a universal instrument for comparing complex systems. EPJ Data Science 12 (1), pp. 37. Cited by: Appendix E, §4.2.

[18] DrissionPage DrissionPage repository. Note: https://github.com/g1879/DrissionPage Cited by: footnote 3. Flesch (2007) R. Flesch Flesch-kincaid readability test.

Retrieved October 26 (3), pp. 2007. Cited by: §6.1. Gao et al.

(2023) T. Gao, H. Yen, J. Yu, and D.

Chen Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K.

Bali (Eds.), Singapore, pp. 6465–6488. External Links: Link, Document Cited by: §1, §2. Gastwirth (1971) J.

L. Gastwirth A general definition of the lorenz curve. Econometrica: Journal of the Econometric Society, pp. 1037–1039.

Cited by: §4.1. Gezici et al. (2021) G. Gezici, A.

Lipani, Y. Saygin, and E. Yilmaz Evaluation metrics for measuring bias in search engine results. Information Retrieval Journal 24 (2), pp.

85–113. Cited by: §3.3. Giorgi et al. (2025) T.

Giorgi, L. Cima, T. Fagni, M. Avvenuti, and S.

Cresci Human and llm biases in hate speech annotations: a socio-demographic analysis of annotators and targets. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 19, pp. 653–670.

Cited by: §1. Guan and Cutrell (2007) Z. Guan and E. Cutrell An eye tracking study of the effect of target rank on web search.

In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 417–420. Cited by: §2. Gunawan et al.

(2021) J. Gunawan, A. Pradeep, D. Choffnes, W.

Hartzog, and C. Wilson A comparative study of dark patterns across web and mobile modalities. Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2), pp. 1–29.

Cited by: §4.3. Haim et al. (2018) M. Haim, A.

Graefe, and H. Brosius Burst of the filter bubble? effects of personalization on the diversity of google news. Digital journalism 6 (3), pp.

330–343. Cited by: §1. Hannak et al. (2013) A.

Hannak, P. Sapiezynski, A. Molavi Kakhki, B. Krishnamurthy, D.

Lazer, A. Mislove, and C. Wilson Measuring personalization of web search. In Proceedings of the 22nd international conference on World Wide Web, pp.

527–538. Cited by: §3.3. Huang et al. (2025) L.

Huang, W. Yu, W. Ma, W. Zhong, Z.

Feng, H. Wang, Q. Chen, W. Peng, X.

Feng, B. Qin, et al. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), pp.

1–55. Cited by: §1. Ieong et al. (2012) S.

Ieong, N. Mishra, E. Sadikov, and L. Zhang Domain bias in web search.

In Proceedings of the fifth ACM international conference on Web search and data mining, pp. 413–422. Cited by: §2. [30] G.

Inc. Google search development document. Note: https://developers.google.com/search Cited by: §1, §2. [31] G.

Inc. Google trend document. Note: https://developers.google.com/search/docs/monitor-debug/trends-start Cited by: Appendix I. [32] G.

Inc. Site map. Note: https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview Cited by: §2. [33] S.

Inc. How much traffic is from organic search: (updated for 2025). Note: https://www.seoinc.com/seo-blog/much-traffic-comes-organic-search/ Cited by: §2, item 1, §4.3, footnote 6. [34] S.

E. Journal Google eeat: what is it & how to demonstrate it for seo. Note: https://www.searchenginejournal.com/google-e-e-a-t-how-to-demonstrate-first-hand-experience/474446/ Cited by: §1, §2. Jun et al.

(2018) S. Jun, H. S. Yoo, and S.

Choi Ten years of research change using google trends: from the perspective of big data utilizations and applications. Technological forecasting and social change 130, pp. 69–87. Cited by: §3.2.

Kabir et al. (2023) S. Kabir, D. N.

Udo-Imeh, B. Kou, and T. Zhang Who answers it better? an in-depth analysis of chatgpt and stack overflow answers to software engineering questions.

CoRR. Cited by: §2. Kassem et al. (2022) M.

M. Kassem, A. Raman, D. Perino, and N.

Sastry A browser-side view of starlink connectivity. In Proceedings of the 22nd ACM Internet Measurement Conference, pp. 151–158. Cited by: §4.3.

Kincaid et al. (1975) J. P. Kincaid, R.

P. Fishburne Jr, R. L. Rogers, and B.

S. Chissom Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report Cited by: §6.1. Landis and Koch (1977) J.

R. Landis and G. G. Koch The measurement of observer agreement for categorical data.

biometrics, pp. 159–174. Cited by: Appendix C, Appendix I. Lastowka (2000) F.

G. Lastowka Search engines, html, and trademarks: what’s the meta for. Va. L.

Rev. 86, pp. 835. Cited by: §6.1.

Le Pochat et al. (2019a) V. Le Pochat, T. Van Goethem, and W.

Joosen Evaluating the long-term effects of parameters on the characteristics of the tranco top sites ranking. In Proceedings of the 12th USENIX Conference on Cyber Security Experimentation and Test, CSET’19, USA, pp. 10. Cited by: §3.4.

Le Pochat et al. (2019b) V. Le Pochat, T. Van Goethem, S.

Tajalizadehkhoob, M. Korczyński, and W. Joosen Tranco: a research-oriented top sites ranking hardened against manipulation. In Proceedings of the 26th Annual Network and Distributed System Security Symposium, NDSS 2019.

External Links: Document Cited by: §3.4. Lewis et al. (2020) P. Lewis, E.

Perez, A. Piktus, F. Petroni, V. Karpukhin, N.

Goyal, H. Küttler, M. Lewis, W. Yih, T.

Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, pp. 9459–9474.

Cited by: §1, §2. Lewis et al. (2021) P. Lewis, E.

Perez, A. Piktus, F. Petroni, V. Karpukhin, N.

Goyal, H. Küttler, M. Lewis, W. Yih, T.

Rocktäschel, S. Riedel, and D. Kiela Retrieval-augmented generation for knowledge-intensive nlp tasks. External Links: 2005.11401, Link Cited by: §1, §2.

Li and Aral (2025) H. Li and S. Aral Human trust in ai search: a large-scale experiment. External Links: 2504.06435, Link Cited by: §5.1.

Lundberg and Lee (2017) S. M. Lundberg and S. Lee A unified approach to interpreting model predictions.

In Advances in Neural Information Processing Systems (NeurIPS), pp. 4765–4774. Cited by: §6.2. Luo et al.

(2025) Z. Luo, Z. Peng, Y. Liu, Z.

Sun, M. Li, J. Zheng, and X. He Unsafe llm-based search: quantitative analysis and mitigation of safety risks in ai web search.

arXiv preprint arXiv:2502.04951. Cited by: §1, §2, §2, §3.2. McKight and Najab (2010) P. E.

McKight and J. Najab Kruskal-wallis test. The corsini encyclopedia of psychology, pp. 1–1.

Cited by: §4.3. Menick et al. (2022) J. Menick, M.

Trebacz, V. Mikulik, J. Aslanides, F. Song, M.

Chadwick, M. Glaese, S. Young, L. Campbell-Gillingham, G.

Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147. Cited by: §1, §2.

Meta AI (2025a) Meta AI Everything we announced at our first-ever llamacon. Note: https://ai.meta.com/blog/llamacon-llama-news/Accessed: 2025-11-27. Announces Llama 4 Maverick (400B MoE, 17B active). Cited by: Table 10.

Meta AI (2025b) Meta AI Everything we announced at our first-ever llamacon. Note: https://ai.meta.com/blog/llamacon-llama-news/Accessed: 2025-11-27. Announces Llama 4 Scout (109B MoE, 17B active). Cited by: Table 10.

Miroyan et al. (2025) M. Miroyan, T. Wu, L.

King, T. Li, J. Pan, X. Hu, W.

Chiang, A. N. Angelopoulos, T. Darrell, N.

Norouzi, and J. E. Gonzalez Search arena: analyzing search-augmented llms. External Links: 2506.05334, Link Cited by: §2, §3.2, §4.1.

Nakano et al. (2021) R. Nakano, J. Hilton, S.

Balaji, J. Wu, L. Ouyang, C. Kim, C.

Hesse, S. Jain, V. Kosaraju, W. Saunders, et al.

Webgpt: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. Cited by: §1, §2. Naveed et al.

(2025) H. Naveed, A. U. Khan, S.

Qiu, M. Saqib, S. Anwar, M. Usman, N.

Akhtar, N. Barnes, and A. Mian A comprehensive overview of large language models. ACM Trans.

Intell. Syst. Technol. 16 (5).

External Links: ISSN 2157-6904, Link, Document Cited by: §1. Nielsen (2016) R. K. Nielsen News media, search engines and social networking sites as varieties of online gatekeepers.

In Rethinking journalism again, pp. 93–108. Cited by: §1. Pan et al.

(2007) B. Pan, H. Hembrooke, T. Joachims, L.

Lorigo, G. Gay, and L. Granka In google we trust: users’ decisions on rank, position, and relevance. Journal of computer-mediated communication 12 (3), pp.

801–823. Cited by: §2, §4.3. Peng et al. (2019) P.

Peng, L. Yang, L. Song, and G. Wang Opening the blackbox of virustotal: analyzing online phishing scan engines.

In Proceedings of the Internet Measurement Conference, IMC ’19, New York, NY, USA, pp. 478–485. External Links: ISBN 9781450369480, Link, Document Cited by: Appendix H, §3.4, §5.2. Pochat et al.

(2018) V. L. Pochat, T. van Goethem, and W.

Joosen Rigging research results by manipulating top websites rankings. CoRR abs/1806.01156. External Links: Link, 1806.01156 Cited by: item 2, §4.3. Poudel and Weninger (2024) A.

Poudel and T. Weninger Navigating the post-api dilemma. In Proceedings of the ACM Web Conference 2024, pp. 2476–2484.

Cited by: Appendix E, §4.2. Poudel and Weninger (2025) A. Poudel and T. Weninger Social and political framing in search engine results.

arXiv preprint arXiv:2507.13325. Cited by: §2. Press et al. (2023) O.

Press, M. Zhang, S. Min, L. Schmidt, N.

A. Smith, and M. Lewis Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.

5687–5711. Cited by: §4.2. Rajaraman and Ullman (2011) A. Rajaraman and J.

D. Ullman Mining of massive datasets. Autoedicion. Cited by: §5.2.

Shah et al. (2025) B. S. Shah, D.

S. Shah, and V. Attar Decoding news bias: multi bias detection in news articles. In Proceedings of the 2024 8th International Conference on Natural Language Processing and Information Retrieval, NLPIR ’24, New York, NY, USA, pp.

97–104. External Links: ISBN 9798400717383, Link, Document Cited by: §3.4. [64] H. Sistemas VirusTotal documentation.

Note: https://docs.virustotal.com/docs/results-reports Cited by: §3.4. Spatharioti et al. (2025) S. E.

Spatharioti, D. Rothschild, D. G. Goldstein, and J.

M. Hofman Effects of llm-based search on decision making: speed, accuracy, and overreliance. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1–15.

Cited by: §1, §2. [66] Statista Generative ai search survey. Note: https://www.statista.com/statistics/1454204/united-states-generative-ai-primary-usage-online-search/ Cited by: §1. Sun et al.

(2023) D. Sun, A. Abzaliev, H. Kotek, C.

Klein, Z. Xiu, and J. D. Williams DELPHI: data for evaluating llms’ performance in handling controversial issues.

In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 820–827. Cited by: §3.2. Tan et al.

(2025) J. Tan, Z. Dou, W. Wang, M.

Wang, W. Chen, and J. Wen Htmlrag: html is better than plain text for modeling retrieved knowledge in rag systems. In Proceedings of the ACM on Web Conference 2025, pp.

1733–1746. Cited by: §6.1. [69] F. ThreatSeeker Forcepoint threatseeker document.

Note: https://help.forcepoint.com/fpone/migration/webtothreatseekerurl/guid-b8ac3928-a64c-4468-bf16-4a6d4932c2c7.html/ Cited by: §3.4. Thurow (2003) S. Thurow Search engine visibility. New Riders.

Cited by: §6.1. Trautwein et al. (2022) D. Trautwein, A.

Raman, G. Tyson, I. Castro, W. Scott, M.

Schubotz, B. Gipp, and Y. Psaras Design and evaluation of ipfs: a storage layer for the decentralized web. In Proceedings of the ACM SIGCOMM 2022 Conference, pp.

739–752. Cited by: §4.3. Trielli and Diakopoulos (2019) D. Trielli and N.

Diakopoulos Search as news curator: the role of google in shaping attention to news information. In Proceedings of the 2019 CHI Conference on human factors in computing systems, pp. 1–15. Cited by: §1.

Trivedi et al. (2023) H. Trivedi, N. Balasubramanian, T.

Khot, and A. Sabharwal Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pp. 10014–10037.

Cited by: §1, §2. Vallina et al. (2020) P. Vallina, V.

Le Pochat, Á. Feal, M. Paraschiv, J. Gamba, T.

Burke, O. Hohlfeld, J. Tapiador, and N. Vallina-Rodriguez Mis-shapes, mistakes, misfits: an analysis of domain classification services.

In Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Cited by: §3.4. Wang et al.

(2016) X. Wang, M. Bendersky, D. Metzler, and M.

Najork Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 115–124. Cited by: §2.

Wazzan et al. (2024) A. Wazzan, S. MacNeil, and R.

Souvenir Comparing traditional and llm-based search for image geolocation. In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, pp. 291–302. Cited by: §2, §3.2.

[77] Wikipedia HITS algorithms. Note: https://en.wikipedia.org/wiki/HITS_algorithm Cited by: §1. [78] Wikipedia Page rank algorithms. Note: https://en.wikipedia.org/wiki/PageRank Cited by: §1, §2.

[79] Wikipedia Tf-idf algorithms. Note: https://en.wikipedia.org/wiki/Tf%E2%80%93idf Cited by: §1, §2. Xu (2000) K. Xu Inference for generalized gini indices using the iterated-bootstrap method.

Journal of Business & Economic Statistics 18 (2), pp. 223–227. Cited by: §4.1. Yang and Menczer (2025) K.

Yang and F. Menczer Accuracy and political bias of news source credibility ratings by large language models. In Proceedings of the 17th ACM Web Science Conference 2025, pp. 127–137.

Cited by: §2. Yang et al. (2018) Z. Yang, P.

Qi, S. Zhang, Y. Bengio, W. Cohen, R.

Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E.

Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, pp.

2369–2380. External Links: Link, Document Cited by: §3.2. Yao et al. (2022) S.

Yao, J. Zhao, D. Yu, N. Du, I.

Shafran, K. R. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models.

In The eleventh international conference on learning representations, Cited by: §1, §2. [84] D. M. V.

Zandt Media bias fact check methodology. Note: https://mediabiasfactcheck.com/methodology/ Cited by: §5.1, §5.1, §5.1. [85] D. M.

V. Zandt Media bias fact check official website. Note: https://mediabiasfactcheck.com Cited by: §3.4, §5.1. Zhou et al.

(2022) D. Zhou, N. Schärli, L. Hou, J.

Wei, N. Scales, X. Wang, D. Schuurmans, C.

Cui, O. Bousquet, Q. Le, et al. Least-to-most prompting enables complex reasoning in large language models.

arXiv preprint arXiv:2205.10625. Cited by: §1, §2. Appendix A Ethics To the best of our knowledge, our dataset contains no personal identifiers or information. To mitigate risks of data misuse, we collect only publicly accessible information and follow ethical guidelines for social data research (Dittrich and Kenneally, 2012).

Our data collection pipeline incorporates rate limiting, exponential backoff, and concurrency controls to avoid impacting the performance of search engines. Furthermore, a waiver was obtained from the institutional ethics committee. Appendix B Bootstrapped Evaluation of the Gini Index To assess differences in inequality across aise and tse, we employed a bootstrap procedure to evaluate Gini indices. For each iteration, we resampled queries with replacement from the observed set of search words for each system, constructed domain count distributions, and computed the corresponding Gini index.

The difference between the two indices was recorded for that replicate. Repeating this process 1,000 times yielded an empirical distribution of Gini differences. From this distribution, we derived a 95% confidence interval by taking the 2.5th and 97.5th percentiles. We also computed the observed difference using the full domain counts, which served as a baseline for comparison.

This non-parametric approach allows us to quantify the robustness of observed differences in domain inequality, mitigating concerns that results may be driven by sampling variability rather than systematic differences between systems. Table 7 reports the statistics results. Table 7. Two-sided bootstrap pp-values for Gini index differences between aise and tse.

Note: * p<0.05p<0.05, ** p<0.01p<0.01, *** p<0.001p<0.001. System Pair ptwo-sidedp_{\text{two-sided}} AI Mode vs Bing T. 0.59 AI Mode vs Google T. 0.58 ChatGPT vs Bing T.

0.103 ChatGPT vs Google T. 0.026 * Copilot vs Bing T. 0.999 Copilot vs Google T. 0.999 Gemini vs Bing T.

0.000 *** Gemini vs Google T. 0.000 *** Grok vs Bing T. 0.000 *** Grok vs Google T. 0.000 *** Perplexity vs Bing T.

0.759 Perplexity vs Google T. 0.672 Appendix C Data Annotation Coverage Rate Table 8 illustrates the coverage rate of each annotation method shown in the main body of our experiments. We evaluate the extent to which each method successfully retrieves categorization data from four distinct external knowledge bases: Forcepoint, Tranco Ranking, mbfc and VirusTotal. Table 8.

Percentages of matched domains across annotations. Forcepoint Tranco Ranking MBFC VirusTotal AI Mode 90% 50% 17% 100% ChatGPT 88% 57% 29% 100% Copilot 78% 47% 16% 100% Gemini 90% 54% 20% 100% Grok 85% 45% 17% 100% Perplexity 89% 50% 17% 100% Google 89% 51% 18% 100% Bing 84% 54% 9% 100% Validations To validate the correctness of existing annotations, we apply a multi-stage human verification to the Forcepoint and VirusTotal labels. Note, we do not perform validation for Tranco or mbfc because neither dataset provides semantic or credibility labels suitable for our verification protocol. Tranco is a domain-ranking list aggregated from multiple traffic–estimation sources; it contains no human- or model-interpretable annotations, only popularity scores.

Since our validation procedure is designed to assess label correctness, Tranco falls outside the scope of label-level verification. Similarly, we omit validation for mbfc, as its assessments are produced directly by trained human reviewers who evaluate outlets based on fact-checking history, content analysis, and editorial practices. Three co-authors independently annotate a randomly selected subset of 100 samples from each (i.e. Forcepoint and VirusTotal) dataset, while remaining blinded to the original labels.

We adopt the same categorization taxonomy as the original third-party annotations to ensure direct comparability. Ground-truth labels are then established using a majority-voting scheme: a label is adopted only if it receives agreement from at least two annotators. Instances that fail to meet this threshold are treated as ambiguous and are subsequently resolved through deliberative group discussion until a final classification is reached. We then calculated Fleiss’ κ\kappa to measure the agreement between the human-verified labels and the original third-party annotations.

The resulting scores of Forcepoint and Virustotal reach 92% and 86%, respectively. These scores (>80%>80\%) indicate a high level of consistency between our manual audit and the external datasets (Landis and Koch, 1977). This statistical evidence confirms that the third-party annotations align with human judgment, thereby validating their accuracy for use in our measurements. Appendix D Overlapped Domains with tse Table 9.

Mean domain overlap between aise and traditional search engines (TSEs). Search Engine Bing (Mean) Google (Mean) ChatGPT 0.041 0.118 Copilot 0.078 0.178 Gemini 0.022 0.120 Grok 0.014 0.055 Perplexity 0.081 0.408 Table 9 presents the mean domain overlap between aise and tse. Values indicate the per response ratio of overlapping domains divided by the number of domains retrieved by Google or Bing. Appendix E Rank Turbulence Divergence (RTD) We compute an RTD score for each domain, with values bounded between 00 and 11.

In all cases, rτ,1r_{\tau,1} denotes the rank derived from tse, while rτ,2r_{\tau,2} represents the rank obtained from aise. To capture directional bias, we adopt a signed formulation: RTD values approaching +1+1 indicate that the domain is more prominently ranked in tse results, whereas values approaching −1-1 signify stronger representation within aise outputs. We count the RTD score pair-by-pair between tse and aise. The RTD calculates the element-wise divergence as follows: (1) |1rτ,1−1rτ,2|\left|\frac{1}{r_{\tau,1}}-\frac{1}{r_{\tau,2}}\right| Here, τ\tau denotes a domain, and rτ,1r_{\tau,1} and rτ,2r_{\tau,2} represent its ranks in lists R1R_{1} and R2R_{2}, respectively.

If a domain appears in only one of the lists, we assign it a rank of max⁡(R)+1\max(R)+1 in the other list to ensure consistent comparison across both corpora. This formulation introduces a bias toward higher-ranked tokens. To control this bias, we introduce a parameter α\alpha: (2) |1rτ,1α−1rτ,2α|1α+1\left|\frac{1}{r_{\tau,1}^{\alpha}}-\frac{1}{r_{\tau,2}^{\alpha}}\right|^{\frac{1}{\alpha+1}} For each domain in the union of R1R_{1} and R2R_{2}, divergence is computed using Equation (2). We set α=13\alpha=\frac{1}{3}, which has been shown to yield a balanced representation (Dodds et al., 2023; Poudel and Weninger, 2024).

Appendix F MBFC Factual Score Transformation The original factual scores from mbfc range from −1-1 (highly factual) to 1010 (lowly factual), which is a nonstandard orientation. To align with conventional political score normalization (ranging from −10-10 to 1010), we apply a linear transformation as follows: (3) Factual Scorenew=10−20​(Factual ScoreMBFC+1)11\text{Factual Score}_{\mathrm{new}}=10-\frac{20(\text{Factual Score}_{\mathrm{MBFC}}+1)}{11} where Factual ScoreMBFC\text{Factual Score}_{\mathrm{MBFC}} is the original MBFC factual score. After transformation, the scale is standardized such that −10-10 denotes the lowest factuality and 1010 denotes the highest factuality. Appendix G Testing the Independence of the MBFC Factual Score (a) (b) Figure 8.

Heatmap for Dunn’s Post-hoc Test (Bonferroni-adjusted p-values) (a) for credibility and (b) for political. All pairwise comparisons are statistically significant. We conducted non-parametric independence tests to examine whether MBFC factual reporting scores, credibility, and political leaning are statistically associated with different search engines. Specifically, we applied the Kruskal–Wallis (KW) test to compare distributions across groups.

For each variable (credibility H=1079054H=1079054,p<0.001p<0.001 and political H=1943828H=1943828, p<0.001p<0.001), the KW test indicated significant differences among search engines. To identify which pairs of search engines differed, we performed post-hoc pairwise comparisons using Dunn’s test with Bonferroni correction. Figure 8 reports the Dunn’s test results on (a)credibility and (b)political. The KW and post-hoc results suggest that search engines differ systematically in the MBFC profiles of the domains they surface, with AISEs more often associated with higher factual reporting scores and credibility, and some exhibiting distinct political lean distributions.

Appendix H Threats Threshold To determine the detection threshold in §5.2, we analyze how many security vendors label each domain in our dataset as malicious or suspicious. Figure 9 shows the empirical cumulative distribution function (ECDF) of vendor tags across all domains. Overall, 3.1% (4,414) of domains are flagged as malicious and 1.2% (2,075) as suspicious by more than one vendor. However, the number of detected malicious domains decreases sharply as the threshold increases (Threshold 2: 0.4% or 646 domains, Threshold 3: 0.1% or 273 domains).

Though a threshold of one positive flag maximizes sensitivity for emerging threats, it risks high false positives due to vendor inconsistencies (Peng et al., 2019). To balance false positives and false negatives, we adopt a stricter threshold of at least two vendor detections, classifying 0.4% (646) of domains as malicious, in line with established practices (Choo et al., 2024; Peng et al., 2019). Figure 9. ECDF of domains by number of security vendors reporting Appendix I Query Categorization Methodology We utilize five open-source llm to categorize the queries.

Table 10 summarizes the model configurations. Each model independently annotates all queries. To ensure direct comparability, we adopt Google’s official category taxonomy (Inc., ). Ground-truth labels are determined via majority voting: a label is accepted only if at least three models agree.

Queries that do not meet this threshold are marked as ambiguous and subsequently resolved through human annotation from one of the authors. For each labeling task, we retrieve the complete set of categories and subcategories from Google’s taxonomy JSON,1414 14 https://trends.google.com/trends/api/explore/pickers/category?hl=en-US parse all 25 top-level categories and their subcategories, and construct the full category list used in our annotation pipeline. Table 10. LLMs used for categorizing the queries Model Family Release Parameters Arch.

Llama 4 Scout (Meta AI, 2025b) 2025.04 109B total/17B active MoE Llama 4 Maverick (Meta AI, 2025a) 2025.04 400B total/17B active MoE Qwen 3 (Alibaba Cloud Qwen Team, 2025) 2025.04 4B Dense Qwen 3 (Alibaba Cloud Qwen Team, 2025) 2025.04 14B Dense DeepSeek-V3 (DeepSeek AI, 2024) 2024.12 671B total/37B active MoE We then replace “{categories_text}” with it in the following prompt, to label the query with 10 tries: Validations We follow the validation methodology described in Appendix C, excluding categories where llm fail to reach consensus (i.e. in such cases, the query categories are annotated by one of the authors). From the categorization dataset, we randomly sample 100 instances and compute Fleiss’ κ\kappa to quantify agreement between the human-verified labels and the original third-party annotations. The resulting κ=0.88\kappa=0.88 reflects a high level of agreement (>0.80>0.80) (Landis and Koch, 1977), indicating that the llm-generated labels are strongly aligned with human judgment and suitable for use in our measurements.

Appendix J Hyperparameter Table 11 illustrates the parameters for our experiments in §6. Table 11. Overview of machine learning models, hyperparameter search space, and best-performing configurations. Model Hyperparameter Grid Best Parameters Logistic Regression C: {0.1, 1, 10}; Penalty: {l2} C=10; Penalty=l2; Solver=lbfgs Random Forest n_estimators: {100, 200}; max_depth: {None, 10, 20} n_estimators=200; max_depth=None XGBoost n_estimators: {100, 200}; max_depth: {3, 6}; learning_rate: {0.05, 0.1} n_estimators=200; max_depth=6; learning_rate=0.1 KNN n_neighbors: {3, 5, 7} n_neighbors=3 MLP Neural Net hidden_layer_sizes: {(50,), (100,)}; alpha: {0.0001, 0.001} hidden_layer_sizes=(100,); alpha=0.001 Appendix K Susceptible Query Category We adopt the category taxonomy from Google Trends to classify each query (see Appendix I).

Figure 10 presents the three most frequent Google Trend categories associated with queries that trigger malicious domains, ranked by their occurrence within each search engine. Figure 10. Distribution of Susceptible Query Categories by Frequency Appendix L HTML Structures • Semantic Tags: This feature measures the total number of semantic HTML elements, including <header>, <nav>, <main>, <article>, <section>, and <footer>. A higher count reflects adherence to modern web development standards that emphasize meaningful structure, improved document hierarchy, and enhanced accessibility.

• Nesting Depth: The nesting depth quantifies the maximum hierarchical level of embedded HTML elements within the Document Object Model (DOM). It captures the structural complexity and organizational depth of a webpage. Greater depth often indicates more elaborate design and layout hierarchy, whereas shallower structures may reflect simpler content organization. • Accessibility Features: This feature represents the number of accessibility-related attributes implemented in the HTML, including alt, role, aria-label, and aria-hidden.

These attributes are essential indicators of a page’s accessibility compliance and its accommodation of assistive technologies. • Markup Errors: The number of deprecated or discouraged HTML tags, such as <font> and <center>, is counted as an indicator of poor or outdated coding practices. A higher frequency of such tags implies lower technical quality and weaker adherence to contemporary web standards.