How do you explain the P-value to a non-data person?

so lets say you test two different versions of UI

now say version A has better numbers compared to version B, now how confident can we be that versionA is superior to VersionB? --> this is where p - value comes in

P values are numbers between 0 and 1  that quantify how confident we can be that versioanA is different than versionB

closer p-value to zero --> more confident that both versions are different

in reality we use 0.05 as threshold 

so if p-value < 0.05 then statistically significant difference

if p>0.05 then statistically no difference

Note: p-value doesnt say the how much different versionA and versionB are, it just says that they are different

Product description:

A P-value is a measure of the probability that an observed difference could have occurred just by random chance.

E.g., Revenue of Company varies which may be depend upon R&D Spend, Administration, Marketing spends, location etc.   

Significance-level (alpha-value): it is the percentage of risk while rejecting the Hypothesis.

If p-value

Objective: improve the Revenue of Company

Brainstorm ideas:

For improving the revenue of company, we will have to decide about the component on which revenue depends.

1.       I will define Significance-level (alpha -value) for the Experiment.

2.       I will build regression model using Revenue as Dependent variables and deciding component as independent variables.

3.      Run the experiment and check p-value of independent variables.

4.      If p-value>alpha-value, eliminate those independent variables.

5.      Again, run the experiment using revenue as Dependent variables and remaining as independent variables. Repeat step 2 to step 4 till we found all p-value

Hypothesis: Revenue of company doesn’t depend upon R&D Spend, Administration, Marketing Spend and location.    

Experiment:

1.       Define significance level (alpha-value)

2.      Build a linear regression model using variables R&D Spend, Administration, Marketing Spend and Location.

3.      Run the Model and check p-value of each variable.

Success Metrics:

1.      P-value of R&D spend

2.      P-value of Administration

3.      P-value of Marketing Spend

4.      P-value of Location

Trade-off:

1.      Although Low p-value promotes the rejection of the hypothesis, it addresses nothing about the probability of rejecting it.

2.      High p-value means that our data is highly consistent with our hypothesis, nothing more.

Once we have found the deciding variables about revenue, we can do A/B test on those variables for improving the revenue.