Eric Dumazet 6c9ae009b2 irq: use per_cpu kstat_irqs ...

Use modern per_cpu API to increment {soft|hard}irq counters, and use
per_cpu allocation for (struct irq_desc)->kstats_irq instead of an array.

This gives better SMP/NUMA locality and saves few instructions per irq.

With small nr_cpuids values (8 for example), kstats_irq was a small array
(less than L1_CACHE_BYTES), potentially source of false sharing.

In the !CONFIG_SPARSE_IRQ case, remove the huge, NUMA/cache unfriendly
kstat_irqs_all[NR_IRQS][NR_CPUS] array.

Note: we still populate kstats_irq for all possible irqs in
early_irq_init().  We probably could use on-demand allocations.  (Code
included in alloc_descs()).  Problem is not all IRQS are used with a prior
alloc_descs() call.

kstat_irqs_this_cpu() is not used anymore, remove it.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Reviewed-by: Christoph Lameter <cl@linux.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

2011-01-13 17:32:31 -08:00

..

debug

Merge branch 'master' into for-next

2010-12-22 18:57:02 +01:00

gcov

llseek: automatically add .llseek fop

2010-10-15 15:53:27 +02:00

irq

irq: use per_cpu kstat_irqs

2011-01-13 17:32:31 -08:00

power

Merge branch 'for-2.6.38/core' of git://git.kernel.dk/linux-2.6-block

2011-01-13 10:45:01 -08:00

time

Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial

2011-01-13 10:05:56 -08:00

trace

Merge branch 'for-2.6.38/core' of git://git.kernel.dk/linux-2.6-block

2011-01-13 10:45:01 -08:00

.gitignore

…

acct.c

pass a struct path to vfs_statfs

2010-08-09 16:48:42 -04:00

async.c

async: use workqueue for worker pool

2010-07-14 11:29:46 +02:00

audit_tree.c

in untag_chunk() we need to do alloc_chunk() a bit earlier

2010-10-30 02:18:32 -04:00

audit_watch.c

audit: make functions static

2010-10-30 01:42:19 -04:00

audit.c

audit: error message typo correction

2010-11-03 13:49:58 -04:00

audit.h

audit: make functions static

2010-10-30 01:42:19 -04:00

auditfilter.c

Audit: add support to match lsm labels on user audit messages

2010-10-30 01:41:57 -04:00

auditsc.c

audit mmap

2010-10-30 08:45:43 -04:00

backtracetest.c

…

bounds.c

kbuild: move bounds.h to include/generated

2009-12-12 13:08:14 +01:00

capability.c

sched: Remove remaining USER_SCHED code

2010-04-02 20:12:00 +02:00

cgroup_freezer.c

cgroup_freezer: update_freezer_state() does incorrect state transitions

2010-10-27 18:03:08 -07:00

cgroup.c

switch cgroup

2011-01-12 20:02:47 -05:00

compat.c

compat: Make compat_alloc_user_space() incorporate the access_ok()

2010-09-14 16:08:45 -07:00

configs.c

llseek: automatically add .llseek fop

2010-10-15 15:53:27 +02:00

cpu.c

Merge branches 'x86-alternatives-for-linus', 'x86-fpu-for-linus', 'x86-hwmon-for-linus', 'x86-paravirt-for-linus', 'core-locking-for-linus' and 'irq-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip

2011-01-06 11:11:50 -08:00

cpuset.c

convert cgroup and cpuset

2010-10-29 04:17:06 -04:00

cred.c

signals: move cred_guard_mutex from task_struct to signal_struct

2010-10-27 18:03:12 -07:00

delayacct.c

headers: taskstats_kern.h trim

2009-09-18 09:48:52 -07:00

dma.c

…

elfcore.c

elf coredump: add extended numbering support

2010-03-06 11:26:46 -08:00

exec_domain.c

sys_personality: remove the bogus checks in sys_personality()->__set_personality() path

2010-08-09 20:45:05 -07:00

exit.c

Merge branch 'perf-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip

2011-01-11 11:02:13 -08:00

extable.c

…

fork.c

Merge branch 'for-2.6.38' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu

2011-01-07 17:02:58 -08:00

freezer.c

Freezer: Fix a race during freezing of TASK_STOPPED tasks

2010-12-24 15:02:40 +01:00

futex_compat.c

futex: Address compiler warnings in exit_robust_list

2010-11-10 13:27:50 +01:00

futex.c

futex: Add futex_q static initializer

2010-11-10 15:01:34 +01:00

groups.c

kernel/groups.c: fix integer overflow in groups_search

2010-09-09 18:57:24 -07:00

hrtimer.c

Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial

2011-01-13 10:05:56 -08:00

hung_task.c

lockup detector: Fix grammar by adding a missing "to" in the comments

2010-08-17 09:11:52 +02:00

hw_breakpoint.c

perf: Dynamic pmu types

2010-12-16 11:36:43 +01:00

irq_work.c

irq_work: Use per cpu atomics instead of regular atomics

2010-12-18 15:54:48 +01:00

itimer.c

itimers: Fix racy writes to cpu_itimer fields

2009-11-18 16:32:12 +01:00

jump_label.c

jump label: Make arch_jump_label_text_poke_early() optional

2010-10-29 12:56:13 -04:00

kallsyms.c

Revert "kernel: make /proc/kallsyms mode 400 to reduce ease of attacking"

2010-11-19 11:54:40 -08:00

Kconfig.freezer

…

Kconfig.hz

…

Kconfig.locks

mutex: Better control mutex adaptive spinning config

2009-12-03 11:50:11 +01:00

Kconfig.preempt

…

kexec.c

tree-wide: fix comment/printk typos

2010-11-01 15:38:34 -04:00

kfifo.c

kfifo: fix scatterlist usage

2010-10-01 10:50:58 -07:00

kmod.c

Make do_execve() take a const filename pointer

2010-08-17 18:07:43 -07:00

kprobes.c

Merge branch 'for-2.6.38' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu

2011-01-07 17:02:58 -08:00

ksysfs.c

sysfs: add struct file* to bin_attr callbacks

2010-05-21 09:37:31 -07:00

kthread.c

sched: Constify function scope static struct sched_param usage

2011-01-07 15:55:45 +01:00

latencytop.c

fs/proc/base.c, kernel/latencytop.c: convert sprintf_symbol() to %ps

2011-01-13 08:03:16 -08:00

lockdep_internals.h

lockdep: No need to disable preemption in debug atomic ops

2010-05-04 05:38:16 +02:00

lockdep_proc.c

locking, lockdep: Convert sprintf_symbol to %pS

2010-11-10 10:23:58 +01:00

lockdep_states.h

…

lockdep.c

lockdep: Check the depth of subclass

2010-10-18 18:44:26 +02:00

Makefile

kernel: clean up USE_GENERIC_SMP_HELPERS

2011-01-13 08:03:08 -08:00

module.c

module: Move RO/NX module protection to after ftrace module update

2010-12-23 09:56:00 -05:00

mutex-debug.c

headers: remove sched.h from interrupt.h

2009-10-11 11:20:58 -07:00

mutex-debug.h

locking: Implement new raw_spinlock

2009-12-14 23:55:32 +01:00

mutex.c

mutexes, sched: Introduce arch_mutex_cpu_relax()

2010-11-26 15:05:34 +01:00

mutex.h

…

notifier.c

sched: Use lockdep-based checking on rcu_dereference()

2010-02-25 10:34:26 +01:00

ns_cgroup.c

cgroup: notify ns_cgroup deprecated

2010-10-27 18:03:09 -07:00

nsproxy.c

include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h

2010-03-30 22:02:32 +09:00

padata.c

Merge git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

2010-08-04 15:23:14 -07:00

panic.c

lib/bug.c: add oops end marker to WARN implementation

2010-08-11 08:59:22 -07:00

params.c

param: locking for kernel parameters

2010-08-11 23:04:20 +09:30

perf_event.c

Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial

2011-01-13 10:05:56 -08:00

pid_namespace.c

include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h

2010-03-30 22:02:32 +09:00

pid.c

Add RCU check for find_task_by_vpid().

2010-08-19 17:18:02 -07:00

pm_qos_params.c

PM / PM QoS: Fix reversed min and max

2010-11-15 22:45:22 +01:00

posix-cpu-timers.c

posix-cpu-timers: Rcu_read_lock/unlock protect find_task_by_vpid call

2010-11-10 13:07:06 +01:00

posix-timers.c

posix-timers: Annotate lock_timer()

2010-10-21 17:30:06 +02:00

printk.c

printk: use RCU to prevent potential lock contention in kmsg_dump

2011-01-13 08:03:09 -08:00

profile.c

llseek: automatically add .llseek fop

2010-10-15 15:53:27 +02:00

ptrace.c

signals: move cred_guard_mutex from task_struct to signal_struct

2010-10-27 18:03:12 -07:00

range.c

kernel/range.c: fix clean_sort_range() for the case of full array

2010-11-12 07:55:31 -08:00

rcupdate.c

Merge branch 'rcu/urgent' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-2.6-rcu into core/rcu

2010-10-07 09:43:11 +02:00

rcutiny_plugin.h

rcu: Distinguish between boosting and boosted

2010-11-29 22:01:56 -08:00

rcutiny.c

rcu: add tracing for TINY_RCU and TINY_PREEMPT_RCU

2010-11-29 22:01:55 -08:00

rcutorture.c

rcu: add priority-inversion testing to rcutorture

2010-10-07 10:41:08 -07:00

rcutree_plugin.h

rcu: increase synchronize_sched_expedited() batching

2010-12-17 12:34:08 -08:00

rcutree_trace.c

rcu,cleanup: simplify the code when cpu is dying

2010-11-29 22:01:58 -08:00

rcutree.c

Merge branch 'for-2.6.38' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu

2011-01-07 17:02:58 -08:00

rcutree.h

rcu: limit rcu_node leaf-level fanout

2010-12-17 12:34:20 -08:00

relay.c

Clean up relay_alloc_page_array() slightly by using vzalloc rather than vmalloc and memset

2010-11-05 08:21:34 -07:00

res_counter.c

include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h

2010-03-30 22:02:32 +09:00

resource.c

resources: add arch hook for preventing allocation in reserved areas

2010-12-17 10:01:09 -08:00

rtmutex_common.h

…

rtmutex-debug.c

sched: Convert pi_lock to raw_spinlock

2009-12-14 23:55:33 +01:00

rtmutex-debug.h

…

rtmutex-tester.c

rtmutex-tester: make it build without BKL

2010-10-19 11:29:56 +02:00

rtmutex.c

rtmutes: Convert rtmutex.lock to raw_spinlock

2009-12-14 23:55:33 +01:00

rtmutex.h

…

rwsem.c

…

sched_autogroup.c

sched: Mark autogroup_init() __init

2011-01-07 15:54:36 +01:00

sched_autogroup.h

sched: Add 'autogroup' scheduling feature: automated per session task groups

2010-11-30 16:03:35 +01:00

sched_clock.c

sched: Add some clock info to sched_debug

2010-11-23 10:29:08 +01:00

sched_cpupri.c

sched: No need for bootmem special cases

2010-07-17 12:06:22 +02:00

sched_cpupri.h

sched: No need for bootmem special cases

2010-07-17 12:06:22 +02:00

sched_debug.c

sched: Add 'autogroup' scheduling feature: automated per session task groups

2010-11-30 16:03:35 +01:00

sched_fair.c

sched: Fix interactivity bug by charging unaccounted run-time on entity re-weight

2010-12-19 16:36:30 +01:00

sched_features.h

sched: Rewrite tg_shares_up)

2010-11-18 13:27:46 +01:00

sched_idletask.c

sched: Cure load average vs NO_HZ woes

2010-04-23 11:02:02 +02:00

sched_rt.c

sched: Implement on-demand (active) cfs_rq list

2010-11-18 13:27:47 +01:00

sched_stats.h

sched_stat: Update sched_info_queue/dequeue() code comments

2010-10-24 13:29:01 +02:00

sched_stoptask.c

sched: Fix cross-sched-class wakeup preemption

2010-11-11 14:37:23 +01:00

sched.c

Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial

2011-01-13 10:05:56 -08:00

seccomp.c

…

semaphore.c

…

signal.c

signals: annotate lock context change on ptrace_stop()

2010-10-27 18:03:12 -07:00

smp.c

kernel: clean up USE_GENERIC_SMP_HELPERS

2011-01-13 08:03:08 -08:00

softirq.c

kernel: clean up USE_GENERIC_SMP_HELPERS

2011-01-13 08:03:08 -08:00

spinlock.c

locking: Cleanup the name space completely

2009-12-14 23:55:33 +01:00

srcu.c

rcu: Make synchronize_srcu_expedited() fast if running readers

2010-11-29 22:02:40 -08:00

stacktrace.c

…

stop_machine.c

stop_machine: convert cpu notifier to return encapsulate errno value

2010-10-26 16:52:15 -07:00

sys_ni.c

powerpc: define a compat_sys_recv cond_syscall

2010-09-23 17:03:55 +10:00

sys.c

kmsg_dump: add kmsg_dump() calls to the reboot, halt, poweroff and emergency_restart paths

2011-01-13 08:03:07 -08:00

sysctl_binary.c

Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial

2011-01-13 10:05:56 -08:00

sysctl_check.c

sysctl: min/max bounds are optional

2010-10-15 14:42:24 -07:00

sysctl.c

sysctl: remove obsolete comments

2011-01-13 08:03:18 -08:00

taskstats.c

taskstats: use better ifdef for alignment

2011-01-13 08:03:19 -08:00

test_kprobes.c

kprobes: Fix selftest to clear flags field for reusing probes

2010-10-14 08:55:27 +02:00

time.c

Kill off a bunch of warning: ‘inline’ is not at beginning of declaration

2010-11-28 23:08:04 +01:00

timeconst.pl

…

timer.c

Merge branch 'timers-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip

2011-01-06 10:42:43 -08:00

tracepoint.c

jump_label: Use more consistent naming

2010-10-18 19:58:56 +02:00

tsacct.c

taskstats: use real microsecond granularity for CPU times

2010-10-27 18:03:17 -07:00

uid16.c

headers: utsname.h redux

2009-09-23 18:13:10 -07:00

up.c

…

user_namespace.c

user_ns: improve the user_ns on-the-slab packaging

2011-01-13 08:03:18 -08:00

user-return-notifier.c

core: Clean up user return notifers use of per_cpu

2009-12-02 10:22:59 +01:00

user.c

fix freeing user_struct in user cache

2010-12-29 11:31:38 -08:00

utsname_sysctl.c

sysctl kernel: Remove binary sysctl logic

2009-11-12 02:04:55 -08:00

utsname.c

utsns: extract creeate_uts_ns()

2009-06-18 13:03:55 -07:00

wait.c

docbook: add more wait/wake/completion to device-drivers docbook

2010-10-26 17:32:41 -07:00

watchdog.c

Merge branch 'for-2.6.38' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu

2011-01-07 17:02:58 -08:00

workqueue_sched.h

workqueue: implement concurrency managed dynamic worker pool

2010-06-29 10:07:14 +02:00

workqueue.c

workqueue: allow chained queueing during destruction

2010-12-20 19:32:04 +01:00