## Logistic regression with tensorflow¶

The 8th KIAS CAC Summer School

Machine Learning Practice I - Logistic Regression example

Author: Yung-Kyun Noh, Ph.D.

Many parts are borrowed from Jiseob Kim's Github.

In [1]:

    import tensorflow as tf

In [2]:

    import numpy as np
    import matplotlib.pyplot as plt

    %pylab inline

    Populating the interactive namespace from numpy and matplotlib

### Data preparation¶

We use a two-class synthetic dataset from two different Gaussian density functions. \begin{eqnarray} p(\mathbf{x}|\mu_c, \Sigma_c) = \frac{1}{\sqrt{2\pi}|\Sigma_c|^\frac{1}{2}} \exp\left(-\frac{1}{2}(\mathbf{x} - \mu_c)^\top\Sigma_c^{-1}(\mathbf{x} - \mu_c)\right) \quad \text{for class} \quad c = {1,2} \end{eqnarray}

The first class uses $\mu_1 = \left(\begin{array}{c} 4 \\ 0 \end{array}\right)$, $\Sigma_1 = \left(\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array}\right)$, and the second class uses $\mu_2 = \left(\begin{array}{c} 0 \\ 4 \end{array}\right)$, $\Sigma_2 = \left(\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array}\right)$.

In [3]:

    n_data_per_class = 50
    num_class = 2
    dim_data = 2
    data1 = np.random.multivariate_normal([4,0], [[1,0],[0,1]], n_data_per_class)
    #print data1
    data2 = np.random.multivariate_normal([0,4], [[1,0],[0,1]], n_data_per_class)
    data_x = np.vstack([data1,data2])
    data_y = np.hstack([np.ones((n_data_per_class,)), -np.ones((n_data_per_class,))])

    def draw_state(wval, bval):
        plt.rcParams['figure.figsize']=(5,5)
        plt.scatter(data1[:,0],data1[:,1],30,'r')
        plt.scatter(data2[:,0],data2[:,1],30,'b')
        [x1min,x1max,x2min,x2max] = plt.axis()
        x1val = np.arange(x1min,x1max,0.1)

        plt.plot(x1val, -(wval[0]*x1val+bval)/wval[1], 'k')
        plt.axis([x1min,x1max,x2min,x2max])
        plt.show()

    W_temp = [1,1]
    b_temp = 0
    draw_state(W_temp, b_temp)

## Logistic regression¶

\begin{eqnarray} P(y = 1|\mathbf{x};\mathbf{w},b) &=& \frac{1}{1 + \exp(-(\mathbf{w}^\top\mathbf{x} + b))} \\ P(y = -1|\mathbf{x};\mathbf{w},b) &=& \frac{1}{1 + \exp(\mathbf{w}^\top\mathbf{x} + b)} \end{eqnarray}

The objective function $L$ with $\mathcal{D}=\{\mathbf{x}_i,y_i\}_{i = 1}^N$ \begin{eqnarray} L &=& -\log P(y_1,\ldots,y_N|\mathbf{x}_1,\ldots,\mathbf{x}_N; \mathbf{w}, b) + \lambda ||\mathbf{w}||^2 \quad \left( P(y_1,\ldots,y_N|\mathbf{x}_1,\ldots,\mathbf{x}N; \mathbf{w}, b) = \prod{i = 1}^N P(y = y_i|\mathbf{x} = \mathbf{x}_i; \mathbf{w}, b) \right) \ &=&-\sum_i \log p(y_i|\mathbf{x}_i; \mathbf{w}, b) + \lambda ||\mathbf{w}||^2 \end{eqnarray}

In [4]:

    # Create the model

    x = tf.placeholder(tf.float32, [None, dim_data])
    #W = tf.Variable(tf.zeros([dim_data,1]))
    W = tf.Variable(tf.ones([dim_data,1]))
    b = tf.Variable(tf.zeros([1]))
    y_true = tf.placeholder(tf.float32, [None])

In [5]:

    a = tf.matmul(x, W) + b
    p_y = tf.sigmoid(tf.mul(a, tf.reshape(y_true, [-1,1]) ))
    print p_y

    Tensor("Sigmoid:0", shape=(?, 1), dtype=float32)

In [6]:

    loss = tf.reduce_mean(-tf.log(p_y))

In [7]:

    train_step = tf.train.GradientDescentOptimizer(0.1).minimize(loss)

In [8]:

    config = tf.ConfigProto()
    config.gpu_options.allow_growth=True
    sess = tf.Session(config=config)
    sess.run(tf.initialize_all_variables())

In [9]:

    # Train
    loss_history = []
    for i in range(10):
        sess.run(train_step, feed_dict={x: data_x, y_true: data_y})
        loss_history.append( sess.run(loss, feed_dict={x: data_x, y_true: data_y}) )
        #print sess.run( p_y, feed_dict={x: data_x, y_true: data_y})
        #print sess.run(W), sess.run(b)

    plt.plot(loss_history)

Out[9]:

    [<matplotlib.lines.Line2D at 0x7f95c03c4e10>]

In [10]:

    draw_state(sess.run(W), sess.run(b))