Running a Linux web server Day 01

with a root user
create #pico /etc/apache2/sites-avaliable/example.org.conf
run     #a2ensite example.org.conf

check the configuration #apache2ctl configtest

run     #apache2ctl graceful
run     #a2enmod userdir
run    #apache2ctl graceful

Install a web application
run    #apt-get install unzip imagemagick

set up mysql database
download coppermine project

cp coppermine project to /var/www/
unzip it and change the name cpg15x to photos
chmod 777 include
chmod albums to 777

CRM and data mining Day 02

Foreign keys must match corresponding PKs to maintain referential integrity.

Surrogate key is a unique value, usually an integer assigned to each row in the dimension. It becomes the PK.

An OLAP engine such as Analysis Services provides a better query language (than SQL) and better computational performance.

Entity Relationship modelling is a technique used to 'abstract' users' data requirements into a model that can be analyzed and ultimately implemented. The objectives of ER modelling are

1. Avoid anomalies, and achieve processing and data storage efficiency by reducing data redundancy (storing data elements once).
2. Provide flexibility and ease of maintenance.
3. Protect the integrity of data by storing it once.

ER modelling and normalization simplify transaction processing as they make transactions as simple as possible (data is stored in one place only). However, normalized databases become very complex, making queries difficult and inefficient. A 'spider-web of joins' is required for many queries.

Every dimensional model is composed of one table with a multi-part key, called the fact table, and a set of smaller tables called dimension tables.

Each dimension table has a single-part primary key that corresponds exactly to one of the components of the multi-part key in the fact table.

A single ER diagram breaks down into multiple Dimensional Model diagrams, or 'stars'.

ER modelling (arguably) does not really model a business,; rather, it models the micro-relationships among data elements.

Facts are the 'verbs or actions' of the business. E.g. taking an order, displaying a web page, printing a book,  handling a customer support request. 

A value in a fact table is a measurement. E.g. quantity ordered, sale amount, call duration.

The level of detail in a fact table is called the grain. It is recommended that the grain be kept to lowest (or finest) level. E.g. one row per sale, one row per service call.

A fact always 'resolves' a many-to-many relationship between the parent (or dimension) tables.

The most useful facts in a fact table are numeric and additive. E.g. sales by month for the last year.

The best way to identify dimensions is to note down every time some one says the word 'by'. For example,
-Sales by manufacturing plant
-Deliveries by method of shipment

"Sales" will become a measure. Manufacturing plant becomes a dimension table.
"Deliveries" becomes a measure. Method of shipment becomes a dimension table.


All foreign keys in the fact table are not allowed NULL.
Measures columns are allowed NULL values.

References

1. FIT5158 Monash University Lecture Notes, 2011

Glossary of terms

Dimensional analysis
Multidimensional conceptual view
Intuitive and high-performance retrieval of data.
High-performance access
Fact tables: contain the measurements associated with a specific business process.
Dimension tables
'Star-like' structure
Star join
Star schema
Dimensional model vs ER model
Multi-part primary key: is a primary key consisting of a set of different keys from dimension tables.
One-to-many relationship
Many-to-many relationship
Many-to-one relationship
Referential integrity
Foreign key (FK)
ETL: extract, transfer and load.
SCD: slowly changing dimensions
Dimensional modelling concepts
Conformed dimensions

CRM and data mining Day 01

The update time: the length of time it takes for a change in the operational data to reflect in the data warehouse. A 24 hour 'wrinkle of time" is recommended for the update time.

low-level granularity  vs. high-level granularity
The lower the level of granularity, the more versatile the query that can be issued.

high-level of detail (the details of every phone call made by a customer for a month) vs. low-level of detail (the summary of phone calls made by a customer for a month)


Granularity: is the level of detail or summarization of the units in the DW. E.g. details of every phone call made by a customer for a month vs the summarized phone calls made by a customer for a month. You can always aggregate detailed data by summarizing, but cannot disaggregate data that is stored only as summaries. This is the benefit of low-level granularity.

Customer relationship management (CRM) promises

1. Faster customer service at lower costs
2. Higher customer satisfaction
3. Better customer retention

Ultimately, these lead to customer loyalty .

Essential questions for business that can be solved by computers (decision support systems).

1. Who are my customers and what products are they buying?
2. Which customers are most likely to go to the competition?
3. What impact will new products/services have on revenue and margins?
4. What product promotions have the biggest impact on revenue?
5. What is the most effective distribution channel?
6. Which are our lowest/highest margin customers?

Organizations need to think about finding products for the customers rather than customers for their products

A key focus of business intelligence is optimizing the lifetime value of customers. To do this we need to

• Get to know the customers better -> create customer profiles
• Interact appropriately with the customers ->

Customer intelligence or 'Analytical CRM' is the process of gathering, analyzing and exploiting information of a company's customer base.

There are 3 main steps:

1. Build a data warehousing for customer intelligence.
2. Use decision support tools (OLAP and reporting)
3. Data mining

Data warehouses enable organizations capture, organize and analyze the information in order to make decisions about the future based on the past. The steps are as follows.

1. Identify the data that needs to be gathered from the operational business systems,
2. Place data in the data warehouse
3. Ask questions of the data to derive valuable information.

Operational information: questions about situations right now

1. How many unfulfilled orders are there?
2. On which items are we out of stock?
3. What is the position of a particular order?

Strategic information is concerned with sales of products over time. A 'mail order wine club' may wish to ask

1. Which product lines are increasing in popularity and which are decreasing?
2. Which product lines are seasonal?
3. Which customers place the same orders on a regular basis?
4. Are some products more popular in different parts of the country?
5. Do customers tend to purchase a particular class of product?

Data warehouse is "A subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making processing" (W.H. Inmon)

Data warehouse is "A copy of transaction data, specifically structured for query and analysis" (Ralph Kimball)

Data warehouse data is a series of snapshots, each snapshot taken at one moment in time.

Data warehouse (DW) data always contains a time element-with entries for different values of time.

References

1. FIT5158 Monash Lecture Notes, 2011

Glossary of terms

Fact table vs. dimension table
Data mart
Star join is an ideal structure for a data mart.
The centre of the star join is called the 'fact table'.
Agility
Integrated computerized decision support
Analysis, decisions, predictions
Margin customers
The lifetime value of customers
Customer-centric businesses
Customer-centric organisations
Elaborate: work out in detail (e.g. elaborate a plan)
Blanket marketing campaigns vs. targeted marketing campaigns
Monetary
OLAP: online analytical processing
OLTP: online transaction processing
A single unified data repository
Repetitive
Decision support system data (DSS data)
Clerical functions vs. managerial functions
Primitive data vs. derived data
Collective time horizon
The volume of data in the warehouse

Questions for the revised Bloom's taxonomy

Learning levels 1, 2, and 3

Learning levels 4, 5, and 6

References
http://www.cccs.edu/Docs/Foundation/SUN/QUESTIONS%20FOR%20THE%20REVISED%20BLOOM.doc

Color image processing

Converting between YCbCr and RGB in MATLAB can be accomplished by the functions rgb2ycbcr and ycbcr2rgb.

References

Oge Marques, Practical Image and Video Processing Using MATLAB, Wiley-IEEE Press, September 2011.

Comics sites

1. http://thisisindexed.com/
2. http://xkcd.com/
3. http://www.applegeeks.com/index.php
4. http://questionablecontent.net/
5. http://www.phdcomics.com/comics.php

Research Day 01 :Writing research plan, Types of research

I. Writing a research plan needs to answer the following.

1. What do you intend to do?
2. Why is the work important?
3. What has already been done?
4. How are you going to do the work?

A Research design is concerned with turning a research question into a testing project. The research design has been considered a "blueprint" for research, dealing with at least four problems: what questions to study, what data are relevant, what data to collect, and how to analyze the results.

Research method: determine a way of doing a systematic investigation to establish facts. It needs to include step-by-step to achieve the goal.

II. Types of Research
1.Quantitative research

• Used to test objective theories by examining the relationship among variables
• Used to quantify the data and generalise the results from the sample to the population of interest
• Used to recommend a final course of action
• Used statistical data analysis

2. Basic research

• Used to understand and generate more knowledge, e.g.
• How to increase the productivity of the employees.
• How to increase the effectiveness of small businesses.
• How to improve the effectiveness of information systems.
• Einstein's Theory of Relativity
• Newton's Contributions

3. Applied research

• Used to find solutions to everyday problems, cure illness, and develop innovative technologies, rather than to acquire knowledge for knowledge's sake.
• Improve agricultural crop production
• Treat or cure a specific disease
• Improve the energy efficiency of homes, offices, or modes of transportation

4. Longitudinal research
5. Qualitative research

• Is primarily exploratory research
• Used to gain an understanding of underlying reasons, opinions, and motivations.
• Provides insights into the problem or helps to develop ideas or hypotheses for potential quantitative research.
• Used to develop an initial understanding
• Used non-statistical data analysis
• Useful for finding detailed information about people's perceptions and attitudes

6. Descriptive research

• Often used to determine characteristics of the target market
• Answers questions like, "who, what, where, when, and how often" questions
• Fact finding investigation

7. Classification research
8. Exploratory research

• Used when the problem is not well-defined
• Used to determine the best research design, data collection method and selection of subjects.
• Used to provide significaion insight into a given situation
• It is not typically generalizable to the population at large
• It relies on secondary research such as reviewing available literature and/or data, or qualitative approaches such as informal discussions with consumers, employees.
• Answers question like: "Why are ticket sales down?"

9. Causal research

• Explores the relationship between two variables
• Answers questions like: "Does increased advertising result in more ticket sales?"

II. Qualitative data vs quantitative data

1. Qualitative data is information about qualities; information that can't be measured, e.g. gender,  softness of skin, softness of a cat, color of the sky, color of your eyes, agree-to-disagree something/opinion/statement

2. Quantitative data: height, weight, blood pressure, shoe size, the length of the fingernails.