Search Engine Architecture

Search is a key navigation tool, and essential for finding a needle in a haystack of data. Indexed data can quickly provide insight and analysis to any data set. Crawling and scraping can aggregate useful data from the web.

Whatever your use case, if you data or website isn’t searchable, then it is probably not as useful as it could be.  Our primary business at NDX is search engine design and architecture. From simple single site searches billion plus document indexes, we have the experience you need to get your search project working.

Indexing – an indexing back end such as Apache Solr or Elastic Search is the heart of a search engine. These complex technologies are key to any fast, relevant search results. We are experts at both, as well as some other methods of indexing data. We are experts as designing, deploying, configuring and managing these systems.

Crawling and Scraping – Crawling and scraping the web can often provide useful data for business analysis, discovery and other needs. We have extensive experience with web scraping technologies such as Apache Nutch, as well as custom built scrapers and bots. We design polite and non aggressive solutions that can help you acquire data while keeping your good name.

Data Import and ETL – Getting your data into one of the above mentioned indexing engines can be a tedious process, and we can quickly design a solution to make your existing MySQL, Oracle, MongoDB or other data searchable.

Front End Design – if your search project requires a graphical front end we can help you design a stylish, functional and fast web interface. We use modern design techniques and can work in most languages and frameworks although we prefer PHP.