Enterprise Search with Elasticsearch

Pulse - Elasticsearch Operations Done Right

On this page

What is Enterprise Search? Components of Elastic Enterprise Search Setting Up Enterprise Search Implementing Workplace Search Implementing App Search Web Crawler Configuration Advanced Features Security Best Practices Performance Optimization Frequently Asked Questions

Enterprise Search with Elasticsearch provides powerful, scalable search capabilities for organizations to make their data easily discoverable. It combines Elasticsearch's search engine with pre-built solutions for workplace content, applications, and websites, enabling organizations to deliver relevant search experiences across all their data sources.

What is Enterprise Search?

Enterprise Search refers to the ability to search across all of an organization's data sources from a single interface. Elastic's Enterprise Search solution provides:

  • Unified search across multiple data sources
  • Relevance tuning for optimal results
  • Security and permissions preservation
  • Analytics and insights into search behavior
  • Customizable search experiences

1. Workplace Search

Search across workplace content from multiple sources:

  • Google Drive
  • Microsoft SharePoint
  • Slack
  • Jira
  • Confluence
  • Salesforce
  • GitHub
  • Custom sources

Key features:

  • Pre-built connectors for 30+ sources
  • Single search interface
  • Document-level permissions
  • Content synchronization
  • Personalized results

2. App Search

Build custom search experiences for applications:

  • Websites
  • Mobile apps
  • E-commerce platforms
  • Documentation sites
  • Knowledge bases

Key features:

  • RESTful API
  • Relevance tuning tools
  • Analytics dashboard
  • Search-as-you-type
  • Faceted search
  • Synonyms and curations

3. Web Crawler

Crawl and index web content:

  • Public websites
  • Internal web applications
  • Documentation sites
  • Knowledge bases

Key features:

  • Configurable crawl rules
  • Scheduled crawling
  • Content extraction
  • Deduplication
  • Robots.txt compliance

Prerequisites

  • Elasticsearch 7.x or 8.x
  • Kibana (same version as Elasticsearch)
  • Sufficient storage for indexed content
  • Network access to data sources

Installation

Using Elastic Cloud

  1. Create Elastic Cloud deployment
  2. Enable Enterprise Search from deployment settings
  3. Access Enterprise Search from Kibana menu
  4. Configure data sources

Self-Hosted Installation

Download Enterprise Search:

curl -L -O https://artifacts.elastic.co/downloads/enterprise-search/enterprise-search-8.x.x.tar.gz
tar -xzf enterprise-search-8.x.x.tar.gz
cd enterprise-search-8.x.x

Configure config/enterprise-search.yml:

allow_es_settings_modification: true

elasticsearch.host: http://localhost:9200
elasticsearch.username: elastic
elasticsearch.password: changeme

kibana.host: http://localhost:5601

secret_management.encryption_keys:
  - secret_key_here

ent_search.external_url: http://localhost:3002

Start Enterprise Search:

bin/enterprise-search

Access URL: http://localhost:3002

Step 1: Configure Content Sources

  1. Navigate to Workplace Search in Kibana
  2. Click Add source or Content Sources
  3. Select source type (e.g., Google Drive, SharePoint)
  4. Authenticate and authorize access
  5. Configure synchronization settings

Step 2: Set Up Permissions

Document-level security:

  • Permissions are synced from source systems
  • Users only see content they have access to
  • Group-based access control
  • Real-time permission updates

Configure user authentication:

# OAuth, SAML, or native authentication
auth.source: elasticsearch-native

# Or integrate with identity providers
auth.source: saml

Step 3: Customize Search Experience

Search interface customization:

  • Logo and branding
  • Color scheme
  • Search filters
  • Result display format

Relevance settings:

  • Content source weights
  • Boost recent documents
  • Field-specific boosting
  • Custom synonyms

Step 4: Enable Analytics

Monitor search usage:

  • Top queries
  • Click-through rates
  • Query refinements
  • Popular content
  • Search trends

Step 1: Create a Search Engine

Via Kibana UI:

  1. Go to Enterprise Search > App Search
  2. Click Create engine
  3. Name your engine
  4. Select language

Via API:

curl -X POST 'http://localhost:3002/api/as/v1/engines' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer private-key' \
  -d '{
    "name": "my-engine",
    "language": "en"
  }'

Step 2: Index Documents

Single document:

curl -X POST 'http://localhost:3002/api/as/v1/engines/my-engine/documents' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer private-key' \
  -d '[{
    "id": "doc-1",
    "title": "Elastic Enterprise Search",
    "body": "Complete guide to Enterprise Search",
    "url": "https://example.com/guide"
  }]'

Bulk indexing:

curl -X POST 'http://localhost:3002/api/as/v1/engines/my-engine/documents' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer private-key' \
  -d '[
    {"id": "doc-1", "title": "Document 1", "body": "Content 1"},
    {"id": "doc-2", "title": "Document 2", "body": "Content 2"}
  ]'

Step 3: Implement Search

Search API request:

curl -X POST 'http://localhost:3002/api/as/v1/engines/my-engine/search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer search-key' \
  -d '{
    "query": "enterprise search",
    "page": {
      "size": 10,
      "current": 1
    },
    "filters": {
      "category": ["documentation"]
    }
  }'

JavaScript client:

import AppSearchClient from '@elastic/app-search-javascript'

const client = AppSearchClient(
  'host-identifier',
  'search-key',
  'my-engine'
)

const options = {
  search_fields: {
    title: { weight: 3 },
    body: { weight: 1 }
  },
  result_fields: {
    title: { raw: {} },
    body: { raw: {} },
    url: { raw: {} }
  }
}

client
  .search('enterprise search', options)
  .then(results => console.log(results))

Step 4: Tune Relevance

Field weights:

  • Assign importance to different fields
  • Boost title vs. body content
  • Configure decay functions

Curations:

  • Pin specific results for queries
  • Hide irrelevant results
  • Promote important content

Synonyms:

{
  "synonyms": [
    ["ES", "Elasticsearch", "Elastic Search"],
    ["documentation", "docs", "guide"]
  ]
}

Boosts and Filters:

{
  "boosts": {
    "recency": {
      "type": "functional",
      "function": "exponential",
      "center": "now",
      "factor": 10
    },
    "popularity": {
      "type": "value",
      "value": [5, 10]
    }
  }
}

Web Crawler Configuration

Setting Up a Crawler

  1. Go to Enterprise Search > App Search > Crawler
  2. Click Add domain
  3. Enter starting URLs
  4. Configure crawl rules

Crawl rules example:

# Include patterns
- type: regex
  pattern: ^https://example\.com/docs/.*
  policy: allow

# Exclude patterns
- type: regex
  pattern: ^https://example\.com/admin/.*
  policy: deny

Extraction Rules

Define how to extract content:

{
  "extraction_rules": [
    {
      "selector": "title",
      "field": "title"
    },
    {
      "selector": "article.content",
      "field": "body"
    },
    {
      "selector": "meta[name='description']",
      "field": "description",
      "attribute": "content"
    }
  ]
}

Scheduling Crawls

Automatic crawling:

  • Set crawl frequency (daily, weekly, etc.)
  • Configure crawl depth
  • Set crawl limits
  • Monitor crawl status

Advanced Features

1. Search UI Components

Pre-built React components:

import {
  SearchProvider,
  SearchBox,
  Results,
  Facet
} from '@elastic/react-search-ui'

function App() {
  return (
    <SearchProvider config={config}>
      <SearchBox />
      <Facet field="category" label="Categories" />
      <Results />
    </SearchProvider>
  )
}

2. Analytics and Insights

Track search performance:

  • Query volume over time
  • Top queries
  • Queries with no results
  • Click-through rate
  • Average position of clicks

API for custom analytics:

curl -X GET 'http://localhost:3002/api/as/v1/engines/my-engine/analytics/queries' \
  -H 'Authorization: Bearer private-key'

3. A/B Testing

Test relevance configurations:

{
  "experiments": [
    {
      "name": "Title Weight Test",
      "variants": [
        {"title_weight": 2},
        {"title_weight": 5}
      ]
    }
  ]
}

4. Machine Learning for Relevance

Adaptive relevance:

  • Learn from click behavior
  • Automatic query refinement
  • Personalized results
  • Continuous optimization

5. Multi-Language Support

Configure language-specific analysis:

{
  "engine": {
    "language": "en",
    "multilingual": true,
    "supported_languages": ["en", "es", "fr", "de"]
  }
}

Security Best Practices

1. API Key Management

Create restricted keys:

# Search-only key (public)
# Admin key (private)
# Analytics key (restricted)

Key rotation:

  • Regular key updates
  • Revoke compromised keys
  • Audit key usage

2. Document-Level Security

  • Sync permissions from source systems
  • Use access control lists (ACLs)
  • Implement role-based access
  • Test permission inheritance

3. Network Security

  • Use HTTPS for all communications
  • Implement IP whitelisting
  • Configure firewall rules
  • Enable audit logging

Performance Optimization

1. Indexing Optimization

Bulk indexing:

  • Batch document uploads
  • Use async processing
  • Monitor indexing rate
  • Handle errors gracefully

Field optimization:

  • Index only searchable fields
  • Use appropriate field types
  • Disable unnecessary features
  • Configure proper analyzers

2. Query Optimization

Caching:

  • Enable query caching
  • Cache frequent queries
  • Set appropriate TTL
  • Monitor cache hit rate

Query design:

  • Use filters when possible
  • Limit result size
  • Implement pagination
  • Use search_fields parameter

3. Scaling

Horizontal scaling:

  • Add more Elasticsearch nodes
  • Increase App Search instances
  • Load balance requests
  • Distribute indices across nodes

Frequently Asked Questions

Q: What's the difference between App Search and Workplace Search?
A: App Search is for building custom search experiences in applications, while Workplace Search is for searching across workplace content sources like Google Drive and Slack.

Q: Can I use Enterprise Search with existing Elasticsearch indices?
A: App Search creates its own indices, but you can use Elasticsearch directly for custom search implementations.

Q: How many documents can Enterprise Search handle?
A: Enterprise Search can scale to billions of documents with proper infrastructure sizing.

Q: Does Enterprise Search support real-time indexing?
A: Yes, documents are searchable within seconds of indexing through the API.

Q: Can I customize the search relevance algorithm?
A: Yes, App Search provides extensive relevance tuning tools including weights, boosts, curations, and synonyms.

Q: Is Enterprise Search available on-premises?
A: Yes, Enterprise Search can be self-hosted or used through Elastic Cloud.

Q: How does Enterprise Search handle permissions?
A: Workplace Search syncs permissions from source systems, ensuring users only see content they have access to.

Q: Can I migrate from other search solutions to Enterprise Search?
A: Yes, Enterprise Search provides APIs for bulk document indexing and supports various data import methods.

Q: What languages does Enterprise Search support?
A: Enterprise Search supports 40+ languages with language-specific text analysis and stemming.

Q: How much does Enterprise Search cost?
A: Enterprise Search pricing depends on deployment size and is included with Elastic Cloud subscriptions or available for self-hosted deployments with appropriate licenses.

Pulse - Elasticsearch Operations Done Right

Pulse can solve your Elasticsearch issues

Subscribe to the Pulse Newsletter

Get early access to new Pulse features, insightful blogs & exclusive events , webinars, and workshops.

We use cookies to provide an optimized user experience and understand our traffic. To learn more, read our use of cookies; otherwise, please choose 'Accept Cookies' to continue using our website.