Data Mining Basics

Definition and Purpose of Data Mining:

Data mining is a relatively new term that refers to the process by which predictive patterns are extracted from information.

Data is often stored in large, relational databases and the amount of information stored can be substantial. But what does this data mean? How can a company or organization figure out patterns that are critical to its performance and then take action based on these patterns? To manually wade through the information stored in a large database and then figure out what is important to your organization can be next to impossible.

This is where data mining techniques come to the rescue! Data mining software analyzes huge quantities of data and then determines predictive patterns by examining relationships.

Data Mining Techniques:

There are numerous data mining (DM) techniques and the type of data being examined strongly influences the type of data mining technique used.

Note that the nature of data mining is constantly evolving and new DM techniques are being implemented all the time.

Generally speaking, there are several main techniques used by data mining software: clustering, classification, regression and association methods.

Clustering:

Clustering refers to the formation of data clusters that are grouped together by some sort of relationship that identifies that data as being similar. An example of this would be sales data that is clustered into specific markets.

Classification:

Data is grouped together by applying known structure to the data warehouse being examined. This method is great for categorical information and uses one or more algorithms such as decision tree learning, neural networks and "nearest neighbor" methods.

Regression:

Regression utilizes mathematical formulas and is superb for numerical information. It basically looks at the numerical data and then attempts to apply a formula that fits that data.

New data can then be plugged into the formula, which results in predictive analysis.

Association:

Often referred to as "association rule learning," this method is popular and entails the discovery of interesting relationships between variables in the data warehouse (where the data is stored for analysis). Once an association "rule" has been established, predictions can then be made and acted upon. An example of this is shopping: if people buy a particular item then there may be a high chance that they also buy another specific item (the store manager could then make sure these items are located near each other).

Data Mining and the Business Intelligence Stack:

Business intelligence refers to the gathering, storing and analyzing of data for the purpose of making intelligent business decisions. Business intelligence is commonly divided into several layers, all of which constitute the business intelligence "stack."

The BI (business intelligence) stack consists of: a data layer, analytics layer and presentation layer.

The analytics layer is responsible for data analysis and it is this layer where data mining occurs within the stack. Other elements that are part of the analytics layer are predictive analysis and KPI (key performance indicator) formation.

Data mining is a critical part of business intelligence, providing key relationships between groups of data that is then displayed to end users via data visualization (part of the BI stack's presentation layer). Individuals can then quickly view these relationships in a graphical manner and take some sort of action based on the data being displayed.


Source: http://ezinearticles.com/?Data-Mining-Basics&id=5120773

Preference to Offshore Document Data Entry Services

A number or business organizations if different industries are seeking competent and precise document data entry services to maintain their business records safe for future references. Document data entry has advanced as a quickly developing and active industry structure almost accept in all major companies of the world. The companies doing businesses these days are undergoing rapid changes and therefore the need for services is becoming all the more crucial.

To get success you need to accomplish more understanding about the market, your business, clients as well as the prevailing factors that influence your business. A considerable amount of document is in one or the other way included in this entire process. These services is helpful in taking crucial decisions for the organization. It also provides you a standard in understanding the current and future business status of your company.

In this information age data-entry from documents and data conversion have become important elements for most business houses. The requirement for document services has reached zenith since companies work on processes like business merger and acquisitions, as well as new technology developments. In such scenarios having access to the right kind of data at the right time is very crucial and that is why companies opt for reliable services.

These services covers a range of professional business oriented activities such as document plus image processing to image editing as well as catalog processing. A few noteworthy examples of from documents include: PDF document indexing, insurance claim entry, online data capture as well as creating new databases. These services are important in industries like insurance companies, banks, government departments and airlines.

Companies such as Offshore and outsource and others offer an entire gamut of first rate data services. Actually, getting services from documents offshore to developing yet competent countries like India has made the process highly economical plus quality driven too.

Business giants around the world have realized multiple advantages associated in Offshore-Data-Entry. Companies not only prosper because of quality services but are also benefited because of better turn around time, maintaining confidentiality of data as well as economic rates.

Though the company works in all form of documents, there are few below mentioned areas where it specializes:

• Document data entry
• Document data entry conversion
• Document data processing
• Document data capture services
• Web data extraction
• Document scanning indexing

Since reputable companies like Offshore Data-Entry hire only well qualified and trained candidates work satisfaction is guaranteed. There are several steps involved in the quality check (QC) process and therefore accuracy level is maintained to 99.995% ensuring that the end result is delivered to the client far beyond his expectation.


Source: http://ezinearticles.com/?Preference-to-Offshore-Document-Data-Entry-Services&id=5570327

Outsource Your Work To Data Entry Services To Convert Your Paperwork To An Electronic Format

Among the many services that are outsourced, data entry services are much in demand. While the job profile might seem simple it does in fact require a certain degree of exactness and an eye for detail. Maintaining and handling the client confidentiality is also very important. Data needs to be processed and the first step is always entering the information in the system. An operator needs to be careful while entering information in the system as often this data is used to collate data and for statistical reports and is also the foundation for all the information on the company. These services include much more than just basic information in this technology driven age. An operator today has projects that require Image entry, card Entry, legal document's entry, medical claim entry, entry for online survey forms, online indexing, copying, pasting and sorting of data etc.

A Data entry operator is competent at handling online as well as offline data and even to excel. Specialized services like Image editing, image clipping and cropping services are also available with this service. BPO companies offer these services at very cost effective rates and the work is processed 24x7 ensuring that the work is constantly auctioned. Many data sensitive projects are also completed even in a 24 hour. There are many online services to choose from and each specializes in various features with ample industry experience. These services use the latest technology to ensure that paperwork is processed in the shorted possible time and is converted into electronic data that is easier to store.

A professional service must be able to offer the following features like data conversion and even storage, effective management of databases and an adherence to turnaround times, 100% accuracy of the data entered, 24x7 webs and phone support, a secure and accurate data capture, data extraction and data processing and importantly a cost effective solution for quality data services. A professional company will also ensure that there is a Quality Assurance department monitoring the quality of the work being handled with relevant feedback to both the client and to the operator.

Before deciding on outsourcing your work to a data entry service ensures that the company is known for its reliability and quality. A company that offers data backup is also a good option as it will take care of all the paperwork while forwarding the converted electronic data back. This paperwork could be extracted in the case of a claim or any legal requirement. There are many BPO companies online advertising their services, browse through their features and find one that suits your requirements.



Source: http://ezinearticles.com/?Outsource-Your-Work-To-Data-Entry-Services-To-Convert-Your-Paperwork-To-An-Electronic-Format&id=7270797

An Easy Way For Data Extraction

There are so many data scraping tools are available in internet. With these tools you can you download large amount of data without any stress. From the past decade, the internet revolution has made the entire world as an information center. You can obtain any type of information from the internet. However, if you want any particular information on one task, you need search more websites. If you are interested in download all the information from the websites, you need to copy the information and pate in your documents. It seems a little bit hectic work for everyone. With these scraping tools, you can save your time, money and it reduces manual work.

The Web data extraction tool will extract the data from the HTML pages of the different websites and compares the data. Every day, there are so many websites are hosting in internet. It is not possible to see all the websites in a single day. With these data mining tool, you are able to view all the web pages in internet. If you are using a wide range of applications, these scraping tools are very much useful to you.

The data extraction software tool is used to compare the structured data in internet. There are so many search engines in internet will help you to find a website on a particular issue. The data in different sites is appears in different styles. This scraping expert will help you to compare the date in different site and structures the data for records.

And the web crawler software tool is used to index the web pages in the internet; it will move the data from internet to your hard disk. With this work, you can browse the internet much faster when connected. And the important use of this tool is if you are trying to download the data from internet in off peak hours. It will take a lot of time to download. However, with this tool you can download any data from internet at fast rate.There is another tool for business person is called email extractor. With this toll, you can easily target the customers email addresses. You can send advertisement for your product to the targeted customers at any time. This the best tool to find the database of the customers.

However, there are some more scraping tolls are available in internet. And also some of esteemed websites are providing the information about these tools. You download these tools by paying a nominal amount.



Source: http://ezinearticles.com/?An-Easy-Way-For-Data-Extraction&id=3517104

How Data Mining Can Help in Customer Relationship Management Or CRM?

Customer relationship management (CRM) is critical activity of improvising customer interactions while at the same time making the interactions more amicable through individualization. Data mining utilizes various data analysis and modeling methods to detect specific patterns and relationships in data. This helps in understanding what a customer wants and forecasting what they will do.

Using Data mining you can find out right prospects and offer them right products. This results in improved revenue because you can respond to each customer in best way using fewer resources.

Basic process of CRM data mining includes:
1. Define business objective
2. Construct marketing database
3. Analyze data
4. Visualize a model
5. Explore model
6. Set up model & start monitoring

Let me explain above steps in detail.

Define the business objective:
Every CRM process has one or more business objective for which you need to construct the suitable model. This model varies depending on your specific goal. The more precise your statement for defining the problem is the more successful is your CRM project.

Construct a marketing database:
This step involves creation of constructive marketing database since your operational data often don't contain the information in the form you want it. The first step in building your database is to clean it up so that you can construct clean models with accurate data.

The data you need may be scattered across different databases such as the client database, operational database and sales databases. This means you have to integrate the data into a single marketing database. Inaccurately reconciled data is a major source of quality issues.

Analyze the data:
Prior to building a correct predictive model, you must analyze your data. Collect a variety of numerical summaries (such as averages, standard deviations and so forth). You may want to generate a cross-section of multi-dimensional data such as pivot tables.

Graphing and visualization tools are a vital aid in data analysis. Data visualization most often provides better insight that leads to innovative ideas and success.



Source: http://ezinearticles.com/?How-Data-Mining-Can-Help-in-Customer-Relationship-Management-Or-CRM?&id=4572272

Data Entry - Outsource or Keep It Local?

Data entry services are usually something of a contentious subject, both in the public's eye and also in the data entry community itself, primarily because of the location from which the services are obtained. Some clients know that a lot of data entry is outsourced to developing countries such as India and Pakistan, and this can often put them off, particularly when the entry is in the English language and many clients would prefer the data inputting to be done by someone whose first language is English.

It's also a raging debate within the data entry industry itself, and there is a conflict between providing well priced services that are outsourced to other countries, or work that is completed locally by native English speakers but who will inevitably charge more.

As an experienced member or a reputable data entry supplier, my experience with outsourced services has been positive for the most part; however, it isn't always the case, much like with anything else in life. Clients are justifiably concerned that the entry work that will come back will not be of an acceptable quality and I'm sure this has happened before.

However there are a few steps that can be taken in order to ensure that the service supplied is not just an adequate one, but an excellent one, and they are mostly common sense, but it's still worthwhile to keep a list of a few things to remember:

- Request a sample

Requesting a sample is always a good idea and not just of any work but of your work itself so that you can see that the work was completed successfully. Any company that knows what they are doing will offer this as a good will gesture as well as to show you that their service is of good quality. This allows you to assess the quality of the output and it may even be a good idea to get a few samples at once just to see what works best for you.

- Check their credentials and experience

There's nothing wrong with start-up companies by any means, but in this case you may want to check their experience and credentials as a company. Here is the UK, it's a very good sign when a data entry company has the ICO, which guarantees higher quality of data entry as well as superior data protection. The ISO 9001:2000 is also one to look out for as this means that the company's work has been approved by the governing body of the industry, the ISO and is checked on regularly.

- Don't be put off by a lack of testimonials

Just because a company doesn't have testimonials doesn't mean they aren't a good company. In this industry, a lot of larger and smaller sized companies for that matter don't always want it to be known who they're outsourcing their work to, let alone that they're outsourcing at all. Therefor it is difficult for companies to come up with a good testimonials list at the best of times.

- Visit or arrange a meeting with the company

Human instinct can be your best friend. Visit the company premises if possible and see if you can see any work being produced or just to check that they are a good establishment that is well set up.

Those are a few steps to take to ensure that you get the right service for you and one that, if you are looking at needing a continuous supply of data entry work completing can become a business partnership to last for years. The argument regarding home-based data entry and outsourcing will go on and on, but the best thing you can do is to trust in the company you choose, make sure to get a sample before committing and judge for yourself when you get (hopefully) great results back.


Source: http://ezinearticles.com/?Data-Entry---Outsource-or-Keep-It-Local?&id=6401739

Why Outsourcing Data Mining Services?

Are huge volumes of raw data waiting to be converted into information that you can use? Your organization's hunt for valuable information ends with valuable data mining, which can help to bring more accuracy and clarity in decision making process.

Nowadays world is information hungry and with Internet offering flexible communication, there is remarkable flow of data. It is significant to make the data available in a readily workable format where it can be of great help to your business. Then filtered data is of considerable use to the organization and efficient this services to increase profits, smooth work flow and ameliorating overall risks.

Data mining is a process that engages sorting through vast amounts of data and seeking out the pertinent information. Most of the instance data mining is conducted by professional, business organizations and financial analysts, although there are many growing fields that are finding the benefits of using in their business.

Data mining is helpful in every decision to make it quick and feasible. The information obtained by it is used for several applications for decision-making relating to direct marketing, e-commerce, customer relationship management, healthcare, scientific tests, telecommunications, financial services and utilities.

Data mining services include:

    Congregation data from websites into excel database
    Searching & collecting contact information from websites
    Using software to extract data from websites
    Extracting and summarizing stories from news sources
    Gathering information about competitors business

In this globalization era, handling your important data is becoming a headache for many business verticals. Then outsourcing is profitable option for your business. Since all projects are customized to suit the exact needs of the customer, huge savings in terms of time, money and infrastructure can be realized.

Advantages of Outsourcing Data Mining Services:

    Skilled and qualified technical staff who are proficient in English
    Improved technology scalability
    Advanced infrastructure resources
    Quick turnaround time
    Cost-effective prices
    Secure Network systems to ensure data safety
    Increased market coverage

Outsourcing will help you to focus on your core business operations and thus improve overall productivity. So data mining outsourcing is become wise choice for business. Outsourcing of this services helps businesses to manage their data effectively, which in turn enable them to achieve higher profits.


Source: http://ezinearticles.com/?Why-Outsourcing-Data-Mining-Services?&id=3066061

Collecting Data With Web Scrapers

There is a large amount of data available only through websites. However, as many people have found out, trying to copy data into a usable database or spreadsheet directly out of a website can be a tiring process. Data entry from internet sources can quickly become cost prohibitive as the required hours add up. Clearly, an automated method for collating information from HTML-based sites can offer huge management cost savings.

Web scrapers are programs that are able to aggregate information from the internet. They are capable of navigating the web, assessing the contents of a site, and then pulling data points and placing them into a structured, working database or spreadsheet. Many companies and services will use programs to web scrape, such as comparing prices, performing online research, or tracking changes to online content.

Let's take a look at how web scrapers can aid data collection and management for a variety of purposes.

Improving On Manual Entry Methods

Using a computer's copy and paste function or simply typing text from a site is extremely inefficient and costly. Web scrapers are able to navigate through a series of websites, make decisions on what is important data, and then copy the info into a structured database, spreadsheet, or other program. Software packages include the ability to record macros by having a user perform a routine once and then have the computer remember and automate those actions. Every user can effectively act as their own programmer to expand the capabilities to process websites. These applications can also interface with databases in order to automatically manage information as it is pulled from a website.

Aggregating Information

There are a number of instances where material stored in websites can be manipulated and stored. For example, a clothing company that is looking to bring their line of apparel to retailers can go online for the contact information of retailers in their area and then present that information to sales personnel to generate leads. Many businesses can perform market research on prices and product availability by analyzing online catalogues.

Data Management

Managing figures and numbers is best done through spreadsheets and databases; however, information on a website formatted with HTML is not readily accessible for such purposes. While websites are excellent for displaying facts and figures, they fall short when they need to be analyzed, sorted, or otherwise manipulated. Ultimately, web scrapers are able to take the output that is intended for display to a person and change it to numbers that can be used by a computer. Furthermore, by automating this process with software applications and macros, entry costs are severely reduced.

This type of data management is also effective at merging different information sources. If a company were to purchase research or statistical information, it could be scraped in order to format the information into a database. This is also highly effective at taking a legacy system's contents and incorporating them into today's systems.

Overall, a web scraper is a cost effective user tool for data manipulation and management.



Source: http://ezinearticles.com/?Collecting-Data-With-Web-Scrapers&id=4223877

Assuring Scraping Success with Proxy Data Scraping

Have you ever heard of "Data Scraping?" Data Scraping is the process of collecting useful data that has been placed in the public domain of the internet (private areas too if conditions are met) and storing it in databases or spreadsheets for later use in various applications. Data Scraping technology is not new and many a successful businessman has made his fortune by taking advantage of data scraping technology.

Sometimes website owners may not derive much pleasure from automated harvesting of their data. Webmasters have learned to disallow web scrapers access to their websites by using tools or methods that block certain ip addresses from retrieving website content. Data scrapers are left with the choice to either target a different website, or to move the harvesting script from computer to computer using a different IP address each time and extract as much data as possible until all of the scraper's computers are eventually blocked.

Thankfully there is a modern solution to this problem. Proxy Data Scraping technology solves the problem by using proxy IP addresses. Every time your data scraping program executes an extraction from a website, the website thinks it is coming from a different IP address. To the website owner, proxy data scraping simply looks like a short period of increased traffic from all around the world. They have very limited and tedious ways of blocking such a script but more importantly -- most of the time, they simply won't know they are being scraped.

You may now be asking yourself, "Where can I get Proxy Data Scraping Technology for my project?" The "do-it-yourself" solution is, rather unfortunately, not simple at all. Setting up a proxy data scraping network takes a lot of time and requires that you either own a bunch of IP addresses and suitable servers to be used as proxies, not to mention the IT guru you need to get everything configured properly. You could consider renting proxy servers from select hosting providers, but that option tends to be quite pricey but arguably better than the alternative: dangerous and unreliable (but free) public proxy servers.

There are literally thousands of free proxy servers located around the globe that are simple enough to use. The trick however is finding them. Many sites list hundreds of servers, but locating one that is working, open, and supports the type of protocols you need can be a lesson in persistence, trial, and error. However if you do succeed in discovering a pool of working public proxies, there are still inherent dangers of using them. First off, you don't know who the server belongs to or what activities are going on elsewhere on the server. Sending sensitive requests or data through a public proxy is a bad idea. It is fairly easy for a proxy server to capture any information you send through it or that it sends back to you. If you choose the public proxy method, make sure you never send any transaction through that might compromise you or anyone else in case disreputable people are made aware of the data.

A less risky scenario for proxy data scraping is to rent a rotating proxy connection that cycles through a large number of private IP addresses. There are several of these companies available that claim to delete all web traffic logs which allows you to anonymously harvest the web with minimal threat of reprisal. Companies such as http://www.Anonymizer.com offer large scale anonymous proxy solutions, but often carry a fairly hefty setup fee to get you going.

The other advantage is that companies who own such networks can often help you design and implementation of a custom proxy data scraping program instead of trying to work with a generic scraping bot. After performing a simple Google search, I quickly found one company (www.ScrapeGoat.com) that provides anonymous proxy server access for data scraping purposes. Or, according to their website, if you want to make your life even easier, ScrapeGoat can extract the data for you and deliver it in a variety of different formats often before you could even finish configuring your off the shelf data scraping program.

Whichever path you choose for your proxy data scraping needs, don't let a few simple tricks thwart you from accessing all the wonderful information stored on the world wide web!


Source: http://ezinearticles.com/?Assuring-Scraping-Success-with-Proxy-Data-Scraping&id=248993

Why Web Scraping Software Won't Help

How to get continuous stream of data from these websites without getting stopped? Scraping logic depends upon the HTML sent out by the web server on page requests, if anything changes in the output, its most likely going to break your scraper setup.

If you are running a website which depends upon getting continuous updated data from some websites, it can be dangerous to reply on just a software.

Some of the challenges you should think:

1. Web masters keep changing their websites to be more user friendly and look better, in turn it breaks the delicate scraper data extraction logic.

2. IP address block: If you continuously keep scraping from a website from your office, your IP is going to get blocked by the "security guards" one day.

3. Websites are increasingly using better ways to send data, Ajax, client side web service calls etc. Making it increasingly harder to scrap data off from these websites. Unless you are an expert in programing, you will not be able to get the data out.

4. Think of a situation, where your newly setup website has started flourishing and suddenly the dream data feed that you used to get stops. In today's society of abundant resources, your users will switch to a service which is still serving them fresh data.

Getting over these challenges

Let experts help you, people who have been in this business for a long time and have been serving clients day in and out. They run their own servers which are there just to do one job, extract data. IP blocking is no issue for them as they can switch servers in minutes and get the scraping exercise back on track. Try this service and you will see what I mean here.



Source: http://ezinearticles.com/?Why-Web-Scraping-Software-Wont-Help&id=4550594

Data Extraction - A Guideline to Use Scrapping Tools Effectively

So many people around the world do not have much knowledge about these scrapping tools. In their views, mining means extracting resources from the earth. In these internet technology days, the new mined resource is data. There are so many data mining software tools are available in the internet to extract specific data from the web. Every company in the world has been dealing with tons of data, managing and converting this data into a useful form is a real hectic work for them. If this right information is not available at the right time a company will lose valuable time to making strategic decisions on this accurate information.

This type of situation will break opportunities in the present competitive market. However, in these situations, the data extraction and data mining tools will help you to take the strategic decisions in right time to reach your goals in this competitive business. There are so many advantages with these tools that you can store customer information in a sequential manner, you can know the operations of your competitors, and also you can figure out your company performance. And it is a critical job to every company to have this information at fingertips when they need this information.

To survive in this competitive business world, this data extraction and data mining are critical in operations of the company. There is a powerful tool called Website scraper used in online digital mining. With this toll, you can filter the data in internet and retrieves the information for specific needs. This scrapping tool is used in various fields and types are numerous. Research, surveillance, and the harvesting of direct marketing leads is just a few ways the website scraper assists professionals in the workplace.

Screen scrapping tool is another tool which useful to extract the data from the web. This is much helpful when you work on the internet to mine data to your local hard disks. It provides a graphical interface allowing you to designate Universal Resource Locator, data elements to be extracted, and scripting logic to traverse pages and work with mined data. You can use this tool as periodical intervals. By using this tool, you can download the database in internet to you spread sheets. The important one in scrapping tools is Data mining software, it will extract the large amount of information from the web, and it will compare that date into a useful format. This tool is used in various sectors of business, especially, for those who are creating leads, budget establishing seeing the competitors charges and analysis the trends in online. With this tool, the information is gathered and immediately uses for your business needs.

Another best scrapping tool is e mailing scrapping tool, this tool crawls the public email addresses from various web sites. You can easily from a large mailing list with this tool. You can use these mailing lists to promote your product through online and proposals sending an offer for related business and many more to do. With this toll, you can find the targeted customers towards your product or potential business parents. This will allows you to expand your business in the online market.

There are so many well established and esteemed organizations are providing these features free of cost as the trial offer to customers. If you want permanent services, you need to pay nominal fees. You can download these services from their valuable web sites also.


Source: http://ezinearticles.com/?Data-Extraction---A-Guideline-to-Use-Scrapping-Tools-Effectively&id=3600918

Data Mining - Retrieving Information From Data

Data mining definition is the process of retrieving information from data. It has become very important now days because data that is processed is usually kept for future reference and mainly for security purposes in a company. Data transforms is processed into information and it is mostly used in different ways depending on what information one is extracting and from where the person is extracting the information.

It is commonly used in marketing, scientific information and research work, fraud detection and surveillance and many more and most of this work is done using a computer. This definition can come in different terms data snooping, data fishing and data dredging all this refer to data mining but it depends in which department one is. One must know data mining definition so that he can be in a position to make data.

The method of data mining has been there for so many centuries and it is used up to date. There were early methods which were used to identify data mining there are mainly two: regression analysis and bayes theorem. These methods are never used now days because a lot of people have advanced and technology has really changed the entire system.

With the coming up or with the introduction of computers and technology, it becomes very fast and easy to save information. Computers have made work easier and one can be able to expand more knowledge about data crawling and learn on how data is stored and processed through computer science.

Computer science is a course that sharpens one skill and expands more about data crawling and the definition of what data mining means. By studying computer science one can be in a position to know: clustering, support vector machines and decision trees there are some of the units that are found on computer science.

It's all about all this and this knowledge must be applied here. Government institutions, small scale business and supermarkets use data.

The main reason most companies use data mining is because data assist in the collection of information and observations that a company goes through in their daily activity. Such information is very vital in any companies profile and needs to be checked and updated for future reference just in case something happens.

Businesses which use data crawling focus mainly on return of investments, and they are able to know whether they are making a profit or a loss within a very short period. If the company or the business is making a profit they can be in a position to give customers an offer on the product in which they are selling so that the business can be a position to make more profit in an organization, this is very vital in human resource departments it helps in identifying the character traits of a person in terms of job performance.

Most people who use this method believe that is ethically neutral. The way it is being used nowadays raises a lot of questions about security and privacy of its members. Data mining needs good data preparation which can be in a position to uncover different types of information especially those that require privacy.

A very common way in this occurs is through data aggregation.

Data aggregation is when information is retrieved from different sources and is usually put together so that one can be in a position to be analyze one by one and this helps information to be very secure. So if one is collecting data it is vital for one to know the following:

    How will one use the data that he is collecting?
    Who will mine the data and use the data.
    Is the data very secure when am out can someone come and access it.
    How can one update the data when information is needed
    If the computer crashes do I have any backup somewhere.

It is important for one to be very careful with documents which deal with company's personal information so that information cannot easily be manipulated.


Source: http://ezinearticles.com/?Data-Mining---Retrieving-Information-From-Data&id=5054887

Data Discovery vs. Data Extraction

Looking at screen-scraping at a simplified level, there are two primary stages involved: data discovery and data extraction. Data discovery deals with navigating a web site to arrive at the pages containing the data you want, and data extraction deals with actually pulling that data off of those pages. Generally when people think of screen-scraping they focus on the data extraction portion of the process, but my experience has been that data discovery is often the more difficult of the two.

The data discovery step in screen-scraping might be as simple as requesting a single URL. For example, you might just need to go to the home page of a site and extract out the latest news headlines. On the other side of the spectrum, data discovery may involve logging in to a web site, traversing a series of pages in order to get needed cookies, submitting a POST request on a search form, traversing through search results pages, and finally following all of the "details" links within the search results pages to get to the data you're actually after. In cases of the former a simple Perl script would often work just fine. For anything much more complex than that, though, a commercial screen-scraping tool can be an incredible time-saver. Especially for sites that require logging in, writing code to handle screen-scraping can be a nightmare when it comes to dealing with cookies and such.

In the data extraction phase you've already arrived at the page containing the data you're interested in, and you now need to pull it out of the HTML. Traditionally this has typically involved creating a series of regular expressions that match the pieces of the page you want (e.g., URL's and link titles). Regular expressions can be a bit complex to deal with, so most screen-scraping applications will hide these details from you, even though they may use regular expressions behind the scenes.

As an addendum, I should probably mention a third phase that is often ignored, and that is, what do you do with the data once you've extracted it? Common examples include writing the data to a CSV or XML file, or saving it to a database. In the case of a live web site you might even scrape the information and display it in the user's web browser in real-time. When shopping around for a screen-scraping tool you should make sure that it gives you the flexibility you need to work with the data once it's been extracted.


Source: http://ezinearticles.com/?Data-Discovery-vs.-Data-Extraction&id=165396